19:02:14 <jeblair> #startmeeting infra
19:02:15 <openstack> Meeting started Tue Jan 13 19:02:14 2015 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:19 <openstack> The meeting name has been set to 'infra'
19:02:23 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
19:02:23 <ianw> o/
19:02:23 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-01-06-19.02.html
19:02:28 <SergeyLukjanov> o/
19:02:30 <krtaylor> o/
19:03:11 <jeblair> several of us are otherwise occupied at LCA, so this may be an abbreviated meeting
19:03:28 <jeblair> #topic Actions from last meeting
19:03:36 <jeblair> asselin update module split staging change and merge asap
19:03:45 <asselin> hi,
19:03:51 <fungi> that's done and merged i believe?
19:03:59 <fungi> i think i approved it anyway
19:04:06 <asselin> Yes, fungi merged it last week
19:04:21 <asselin> I added a sprint here (looking for link)
19:04:29 <jeblair> good!  so we should be in a good place for the sprint on jan 28
19:04:47 <asselin> #link https://wiki.openstack.org/wiki/VirtualSprints#Schedule_of_Upcoming_OpenStack_Virtual_Sprints
19:05:01 <fungi> awesome--thanks asselin!
19:05:03 <asselin> Just need a time & duration
19:05:24 <jeblair> 24 hours?
19:05:40 <asselin> Should be good
19:05:44 <krtaylor> asselin, great! how can we help, are you going to have an example?
19:06:05 <asselin> krtaylor, yes, I have an etherpad to track it
19:06:13 <asselin> #link https://etherpad.openstack.org/p/puppet-module-split-sprint
19:06:16 <jeblair> want to start at 1500 utc?  or earlier if fungi or SergeyLukjanov are up for it?
19:06:30 <SergeyLukjanov> jeblair, I think yup
19:06:38 <fungi> cool with me for 1500 or earlier
19:06:44 <asselin> I'm in california...need to convert it
19:07:02 <asselin> and wondering who else is interested to participate
19:07:23 <Daviey> Is there a theme or list yet?
19:07:31 * jesusaurus wants to participate
19:07:42 <jeblair> Daviey: it's a single-topic sprint to get all our puppet modules split
19:07:50 <asselin> we should be set, but I'd like to do a few practice ones to make sure everything is smooth
19:07:53 <Daviey> jeblair: thanks
19:08:02 <asselin> I have recheck-watch ready
19:08:04 <SergeyLukjanov> jeblair, just to ensure - Jan 28 1500 utc?
19:08:09 <jeblair> SergeyLukjanov: yep
19:08:19 <jeblair> would someone line to send an announcement out to -infra and -dev about that?
19:08:23 <jeblair> like
19:08:33 <SergeyLukjanov> I can do it
19:08:44 <jeblair> SergeyLukjanov: cool, thanks
19:08:57 <jeblair> #action SergeyLukjanov send announcement about puppet module split to infra and dev
19:09:12 <jeblair> asselin: will you update the wiki with the time details?
19:09:19 <asselin> jeblair, will do
19:09:22 <krtaylor> I also added it to the third-party wg agenda
19:09:36 <jeblair> krtaylor: ooh thanks!
19:09:36 <asselin> SergeyLukjanov, please ask ppl who are interested to put their names in the etherpad
19:09:57 <SergeyLukjanov> ack
19:10:34 <asselin> sprint page updated. thanks!
19:10:57 <jeblair> cool, anything else about this topic?
19:10:58 <asselin> SergeyLukjanov, actually this link is probably best: https://wiki.openstack.org/wiki/VirtualSprints#OpenStack_puppet_module_split
19:11:16 <asselin> jeblair, nope thanks
19:11:32 <SergeyLukjanov> asselin, yeah, it's a good endpoint for folks who're interested in it
19:12:03 <jeblair> i think we'll skip swift logs, and we just covered puppet module split...
19:12:13 <jeblair> anyone here to talk about nodepool dib?
19:12:34 <fungi> clarkb: was out on an errand but i have mostly working knowledge of its current state
19:12:52 <jeblair> #topic Priority Efforts: Nodepool DIB
19:13:08 <fungi> at this point image builds seem to be working in hpcloud and we're working through options for dealing with cache update concurrency problems
19:13:24 <jeblair> what's cache update concurrency problems in a nutshell?
19:13:47 <fungi> we can build one image at a time fine, but if they launch at the same time they try to update some of the same things in the common cache and then all but one choke on file locks
19:14:12 <fungi> #link https://review.openstack.org/146925
19:14:16 <fungi> is one proposed workaround
19:14:19 <yolanda> fungi, but images are built based on a queue, right?
19:14:30 <jeblair> yeah, that was my understanding
19:14:32 <fungi> yolanda: they're updated in parallel
19:14:32 <yolanda> so it's sequential build
19:14:48 <fungi> sequential image creation is something we discussed as an alternative
19:14:58 <fungi> since they are relatively fast compared to snapshot image builds
19:14:59 <jeblair> fungi: as i understand it, that's the implementation
19:15:47 <fungi> huh, well if that's the intended implementation it's not how it's currently working. three diskimage creation runs start more or less simultaneously and one of them succeeds while the other two fail on file lock errors
19:15:50 <jeblair> fungi: i mean, this is kind of embarassing, because i reviewed a lot of that code, and i thought, "great, this looks like a swell serial queue system for building images"
19:16:17 <jeblair> fungi: did we accidentally do a builder per-provider or something like that?
19:16:33 <fungi> i'll regroup with clarkb when he gets back and see if we can untangle that
19:16:51 <fungi> i mean, it's one of the "solutions" we were talking about implementing for it, so sounds like it has some support ;)
19:17:14 <yolanda> i've been running that for a while and it just runs one dib process each time, this may be some use case not working fine
19:17:19 <jeblair> okay.  and yeah, i wasn't anticipating this problem, but i was anticipating that the nodepool server had finite resources and we would not be able to build many images at once
19:17:40 <jeblair> so that was the basis for wanting to start with serialized
19:17:59 <clarkb> it may be a separate issue with the cache
19:18:08 <jeblair> fungi, yolanda, clarkb: thanks for looking into it
19:18:08 <fungi> stale locks?
19:18:14 <clarkb> permissions or othe filesystem issues
19:18:32 <fungi> i'll double-check the image update logs to make sure they're running concurrently like we're suspecting
19:18:48 <fungi> but yeah, maybe we jumped to conclusions there
19:19:20 <jeblair> #topic Upgrading Gerrit (zaro)
19:19:26 <jeblair> zaro: how's it going?
19:19:44 <zaro> it's going alright.
19:20:14 <SergeyLukjanov> is it update to 2.9?
19:20:18 <zaro> just realized that newer versions of gerrit need a bouncy castle jar that's no availble on precies
19:20:26 <zaro> SergeyLukjanov: yes
19:20:52 <zaro> so we need to upgrade the gerrit server to trusty
19:21:13 <jeblair> i think that's a fine thing to do anyway
19:21:16 <fungi> i can go ahead and work on building a new review-dev running on trusty this afternoon or tomorrow
19:21:17 <zaro> asked fungi to switch review-dev to a trusty server.  just waiting for that. now
19:21:51 <fungi> zaro: can you handle getting the git repos copied over?
19:22:11 <fungi> the initial server build _should_ go quickly but it's not self-bootstrapping
19:22:19 <zaro> also zuul-dev seems to be out of wack so fungi reset it for me, but wanted to test review-dev (trusty) with zuul-dev, so just waiting for new review-dev before proceeding further.
19:22:41 <fungi> and having a written log of what we had to copy to the new server for review-dev will help us put together a similar trusty migration plan for production
19:23:16 <zaro> fungi: sure
19:23:49 <jeblair> cool, thanks!
19:24:01 <jeblair> #topic  Options for procedural -2 (jeblair)
19:24:17 <jeblair> #link https://etherpad.openstack.org/p/procedural_-2
19:24:59 <jeblair> we've gotten requests for the ability for cores to indicate that a patch will not (must not) merge for temporary procedural reasons
19:25:15 <jeblair> eg, feature freeze, or in our own case, "waiting for scheduled maintenance window"
19:25:45 <jeblair> i did some brainstorming based on what we currently have available in gerrit
19:26:00 <jeblair> and i'm not happy with anything we can do in our current version
19:26:11 <jeblair> you can see what i came up with and my reasons on the etherpad
19:26:36 <jeblair> i tend to think that the best resolution is probably a gerrit plugin, and maybe the WIP plugin is actually all we need
19:26:55 <jeblair> (or if we want to extend it with another state, that could work too)
19:27:12 <fungi> yeah, i agree with everything you've written there. we need a sticky status for a change, so a plugin seems necessary
19:27:30 <fungi> everything else has side effects
19:27:48 <jeblair> so how about this?  mull this over a bit, and see if anyone comes up with any other ideas and maybe next week we decide on something?
19:27:55 <zaro> i think WIP plugin should do it because i believe it introduces a state.
19:28:07 <fungi> sounds good
19:28:13 <jeblair> zaro: can you look into the current status of the wip plugin and find out if it is ready for use (in either 2.8 or 2.9?)
19:28:26 <zaro> definately only in 2.9+
19:28:46 <jeblair> kk.  good thing someone is already working on that.  :)
19:28:54 <zaro> but can also test it against 2.9 myself.
19:29:03 <zaro> just to make sure it's in a good state still
19:29:33 <jeblair> #topic Fedora 21 (ianw 6/1)
19:29:48 <ianw> just a few reviews there to ping on
19:30:05 <ianw> i think we can just replace the f20 d-i-b build in nodepool with f21 at this point
19:30:23 <ianw> a quick one for centos image builds (https://review.openstack.org/145959)
19:30:27 <jeblair> nothing is using f20?
19:30:38 <ianw> not d-i-b nodes
19:30:48 <jeblair> got it.  makes sense.
19:31:23 <ianw> also review to switch the devstack tests to f21; experimental jobs show test is working and all devstack bits merged -> https://review.openstack.org/146760
19:32:05 <ianw> i'll move any remaining f20 users with the hope of removing f20 nodes and just have f21
19:32:38 <ianw> that's all
19:33:02 <ianw> also, in the last minute, sdague approved that one above for f21 devstack jobs :)
19:33:22 <fungi> cool
19:33:27 <ianw> oh and just a ping for -> https://review.openstack.org/146822
19:33:32 <jeblair> #topic Open Discussion
19:33:36 <ianw> Adding devstack-plugin-glusterfs project to stackforge
19:33:50 <jeblair> ah, first devstack-plugin-project
19:33:51 <ianw> it hasn't been out long, but these guys are really keen to get their devstack plugin glusterfs working
19:34:30 <ianw> yeah, if anyone didn't see it, i wrote up https://review.openstack.org/146679 about devstack-plugins
19:34:42 <fungi> there's been some acknowledgement from barry warsaw about the state of our python 3.4 ubuntu bugs, though he's waiting on matthias klose to get back from travelling to address them
19:34:56 <fungi> thanks to dhellmann for pinging barry via e-mail
19:36:02 <jeblair> fungi: oh good, thanks you and dhellmann!
19:36:08 <fungi> oh, and i moved paste.openstack.org back to a local database, so keep an eye out for any (old or new) issues with it
19:36:29 <jeblair> fungi: thanks for that.  and :(
19:36:32 <pleia2> just a quick zanata update, making progress but ran into some sections that are undocumented setup-wise, so I'm currently working with camunoz of redhat to fill in those gaps
19:36:44 <pleia2> I think we're the first team outside of redhat to deploy this on our own, so growing pains :)
19:36:57 <fungi> and as zaro mentioned, i rebuilt zuul-dev so if you have anything on the old filesystem i haven't deleted the instance yet but will likely do so soon. fair warning
19:38:01 <asselin> I'm working on an in-tree 3rd party ci spec. I have lots of +1s for "yes this is a good idea". I'd like some more opinions on the best way to structure it and get there. https://review.openstack.org/#/c/139745/3/specs/thirdpartyci.rst
19:38:08 <jeblair> pleia2: thanks!
19:38:21 <jeblair> fungi: ack.  nothing here
19:39:06 <jeblair> asselin: +1 good idea! :)  it'll probably be a little bit before i can give it the attention i need to
19:39:56 <asselin> ok
19:40:59 <fungi> oh, and krotscheck brought up a great point with using docs-draft for storyboard-webclient changes with storyboard-dev on https with a self-signed cert. ajax means cross-domain javascript protections will magically break without manual browser-side preparation
19:41:03 <asselin> for anyone else interested or experience with puppet and/or system-config project, take a look and let me know. thanks!
19:41:32 <fungi> but then we realized that docs-draft is serving the ui via http, so decided that it should be perfectly safe to do http for the api on storyboard-dev
19:41:41 <fungi> so that's the current plan there
19:42:10 <krotscheck> fungi: Speaking of which, do you want to do that or should i?
19:42:39 <fungi> krotscheck: if you want to have a go at it, great. if not, i'll probably get to it but not quickly
19:43:00 <krotscheck> fungi: wfm
19:43:04 <krotscheck> I’ll put it on the list
19:43:19 <fungi> thanks!
19:44:09 <jeblair> thanks everyone!
19:44:14 <jeblair> #endmeeting