19:02:11 <jeblair> #startmeeting infra
19:02:12 <openstack> Meeting started Tue Mar 18 19:02:11 2014 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:16 <openstack> The meeting name has been set to 'infra'
19:02:19 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
19:02:23 <jeblair> agenda ^
19:02:27 <jeblair> #link http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-03-11-19.01.html
19:02:29 <jeblair> last meeting ^
19:02:37 <fungi> yo
19:02:50 <anteaya> o/
19:02:52 <jeblair> #topic  Actions from last meeting
19:03:05 <jeblair> oh that was easy
19:03:09 <nibalizer> o/
19:03:24 <jeblair> #topic  Upgrade gerrit (zaro)
19:04:04 <jeblair> zaro: do you have the etherpad handy for this?
19:04:08 <zaro> changes waiting to be reviewed.
19:04:15 <zaro> ohh.  let me find it.
19:04:31 <jeblair> zaro: i'd like to check in on what changes we need to review now, and which ones we should be planning on merging when we actually do the upgrade
19:05:20 <zaro> https://etherpad.openstack.org/p/gerrit-2.8-upgrade
19:05:25 <jeblair> #link https://etherpad.openstack.org/p/gerrit-2.8-upgrade
19:05:53 <zaro> here is the list
19:06:01 <zaro> #link https://review.openstack.org/#/c/79107
19:06:19 <zaro> #link https://review.openstack.org/#/c/70818/
19:06:32 <zaro> #link https://review.openstack.org/#/c/60893/
19:06:45 <zaro> #link https://review.openstack.org/#/c/60080/
19:06:59 <zaro> #link https://review.openstack.org/#/c/69768/
19:07:15 <zaro> #link https://review.openstack.org/#/c/70014/
19:07:31 <zaro> #link https://review.openstack.org/#/c/70864/
19:07:36 <jeblair> zaro: this week, can you update the etherpad and put those in 3 sections?
19:07:49 <jeblair> zaro: the changes we need to merge now, during the upgrade, and after the upgrade?
19:08:18 <zaro> sure. but that is all the remaining.
19:08:27 <jeblair> zaro: because it seems like some of those may not be safe to merge right now, right?
19:08:42 <zaro> maybe 2 are not, but rest can.
19:09:03 <zaro> i'll put it on etherpad and send out to irc.
19:09:14 <jeblair> zaro: coool thanks.
19:09:53 <zaro> ohh question.
19:10:11 <zaro> should we upgrade during openstack 'off' week?
19:10:24 <clarkb> I won't be around that week
19:10:48 <jeblair> #link https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
19:10:53 <zaro> ohh yeah.  i think clarkb was actually suggesting week before if we can gett all changes approved.
19:10:56 <clarkb> I was thinking week before might be slightly better as only fungi will be out that week iirc
19:11:02 <anteaya> the off week is designed for infra to be able to take a week off
19:11:08 <clarkb> anteaya: exactly :)
19:11:09 <anteaya> at least that was the intend I heard
19:11:16 <jeblair> clarkb: the week of 24?
19:11:30 <clarkb> jeblair: ya
19:11:49 * fungi will be missing in action for the two weeks prior to the off week, and then will be around for the off week, bizarrely
19:11:52 <clarkb> I get on a plane on the 26th (don't let me stop anyone from doing it during the off week fi you want to do it then)
19:11:54 <jeblair> anteaya: i don't think that was the intent -- i think the intent was that everyone should take a week off.  it's more about release schedule timing than giving infra a break.
19:12:02 <jeblair> fungi: i will be around during the off week
19:12:10 <clarkb> jeblair: anteaya right it was an everyone off week including infra
19:12:17 <anteaya> oh I thought ttx thought infra needed to be able to take a break
19:12:46 <anteaya> and that a week needed to be scheduled since infra, release management and qa seem to work when everyone else is off
19:12:58 <fungi> i think ttx just needed to be able to go skiing ;)
19:13:03 <jeblair> anteaya: no, it's because the time between summits was too short for another milestone and too long for one less
19:13:10 <anteaya> oh okay
19:13:22 <jeblair> anteaya: and the time between the release and the summit is unstructured
19:13:42 <jeblair> anteaya: so having that time be too long means people are working on things that may not have been discussed or approved
19:13:44 * zaro is not off @ off week.
19:13:46 <jeblair> so we want to minimize it
19:14:37 <jeblair> zaro: let's see if a few more people will be around; if so, i like the idea of doing it during the off week
19:14:59 <pleia2> I'm around too
19:15:24 <clarkb> huh I figured everyone would be taking advantage of the off week
19:15:26 <fungi> i'm totally up for that
19:15:33 <fungi> people come back from break to a new gerrit ;)
19:15:47 <pleia2> clarkb: I still have some of my humanity, I don't plan my *whole* life around the openstack release cycle :)
19:15:48 <mordred> oh.
19:15:50 <mordred> o/
19:15:53 <mordred> I'm actually awake still
19:16:03 <jeblair> mordred: have plans for the off week?  https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
19:16:10 <jeblair> mordred: we're thinking of doing the gerrit upgrade then
19:16:11 <fungi> pleia2: we'll fix that soon, i promise
19:16:13 <clarkb> pleia2: it actually worked out great for me so win for me I guess
19:16:16 <pleia2> fungi: haha
19:16:33 <mordred> jeblair: works for me
19:16:34 <pleia2> I'm only gone for the weekend (conference)
19:17:01 <jeblair> clarkb, fungi: i'm going to be tooling around NC the week between the off week and summit (and then road tripping to the summit)
19:17:15 <jeblair> clarkb: so that's great that you'll be around to fix any problems that crop up then! ;)
19:17:25 <anteaya> I'm around that week
19:17:35 <fungi> jeblair: i'm around that week too, so should be fine
19:17:36 <clarkb> jeblair: ha
19:18:00 <fungi> jeblair: we'll be running a whole new fork of gerrit when you get back ;)
19:18:54 <jeblair> fungi: and after my next vacation, vinz? :)
19:19:02 <fungi> likely
19:19:11 <pleia2> :)
19:19:13 <clarkb> no no first pygit needs to be fixed :)
19:19:37 <jeblair> ok, so let's tentatively say we'll upgrade during the off week....
19:19:47 <clarkb> wfm
19:19:55 <jeblair> zaro: do you want to send a note to the ml suggesting that and see if anyone screams?
19:20:05 <zaro> will do.
19:20:23 <jeblair> if no one does, we can start doing more formal announcements closer to then
19:20:28 <zaro> ohh is that openstack-infra ml or opestack ml?
19:20:35 <clarkb> openstack-dev
19:20:39 <zaro> ok
19:20:42 <clarkb> is probably the best place to get a tempurature
19:21:20 <jeblair> #topic  determining atc for the purposes of ptl and tc elections (anteaya)
19:21:45 <fungi> ooh!
19:21:46 <anteaya> o/
19:22:00 <mordred> look. it's an anteaya
19:22:05 <jeblair> anteaya: welcome back!
19:22:11 <fungi> anteaya: we run a script to generate that currently, from the programs list in the gobernance repo
19:22:14 <jeblair> in the agenda, anteaya says " does using projects.yaml from the governance repo get us an accurate list? "
19:22:16 <fungi> governance
19:22:19 <anteaya> so the governance repo has a handy projects.yaml: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml
19:22:21 <jeblair> #link http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml
19:22:28 <anteaya> jeblair: thanks
19:22:53 <fungi> anteaya: that is the list we use. are you saying you believe it to be inaccurate?
19:23:02 <anteaya> so what would happen if that was used as a reference for atc list generation for voters
19:23:19 <anteaya> fungi: no, just wanting to ensure I am using it the correct way
19:23:25 <fungi> yep
19:23:27 <anteaya> for instance Barbican is on that list
19:23:41 <anteaya> but to my knowledge is not an integrated project
19:23:43 <fungi> when did barbican join a tc-approved program?
19:23:50 <fungi> right
19:23:53 <anteaya> that was my question
19:23:54 <clarkb> git log?
19:23:58 <mordred> it's incubated
19:24:05 <mordred> incubated counts for atc
19:24:13 <anteaya> when did it get incubated?
19:24:16 <fungi> that is the list official openstack programs use for determining electorate, and the foundation uses to determine who's an active technical contributor for an official project
19:24:18 <anteaya> I must have missed that
19:24:42 <fungi> and since it's in code review, anyone who spots an error can propose a fix and let the tc vote to approve the update
19:24:51 <jeblair> #link https://review.openstack.org/#/c/77647/
19:24:55 <anteaya> great, so I will use it for electorate generation and evaluation
19:25:30 <anteaya> March 10
19:25:33 <anteaya> jeblair: thanks
19:25:44 <anteaya> okay, my question is answered, thank you
19:25:47 <jeblair> so, looks like it's time to start moving barbican to openstack/
19:26:11 <fungi> sounds like it. maybe climate can settle on a new name asap and we can rename them all at the same shot?
19:26:26 <clarkb> wait more renaming?
19:26:29 <jeblair> fungi: i'll add both to the agenda
19:26:52 <clarkb> should we add a review thing to new project addtions that includes 'googled project name and function didn't see any relevant hits"
19:27:18 <zaro> do all incubated projects live in stackforge?
19:27:25 <jeblair> zaro: no, they live in openstack/
19:27:45 <jeblair> zaro: much easier to move them when the move into incubation than when they graduate
19:28:04 <jeblair> also, they become official projects at that point, so it makes sense
19:28:05 <pleia2> pre-incubation they typically are in stackforge
19:28:21 <ttx> (late remark: the idea behind the off week is that 3 weeks is way too much time between release and summit, so we might as well try to encourage a vacation week in there)
19:28:22 <jeblair> pleia2: ++ and actually the tc has made that a requirement now
19:28:29 <pleia2> jeblair: ah, nice
19:28:36 <zaro> so what's the logical seperation between incubated and official? is does none exist?
19:28:38 <jeblair> ttx: we interpreted "vacation" as "upgrade gerrit"  :)
19:28:48 <clarkb> except for me
19:28:53 <fungi> right, at least when they're still only incubated, there aren't usually cross-dependencies requiring coordinated changes to how we're testing any official projects during the rename
19:28:55 <anteaya> fungi: the script that runs on the governance/projects.yaml it uses the preferred email from gerrit, yes?
19:28:59 * clarkb is going to vacation in hawaii that week
19:29:01 <pleia2> zaro: repo organization-wise, none really
19:29:07 <anteaya> yay clarkb
19:29:11 <ttx> jeblair: whatever you like to do as vacation !
19:29:18 <jeblair> zaro: incubated is a subset of official (integrated is also a subset of official)
19:29:42 <fungi> anteaya: yes, we run a database query locally on the gerrit server's backend and then feed that list of addresses and activity into the script
19:29:43 * ttx will be in Barcelona
19:29:45 <jeblair> zaro: incubated and integrated are disjoint subsets of official
19:30:21 <anteaya> fungi: thought so, just wanted to confirm since I will be instructed electorate to confirm the gerrit prefered address in election prep
19:30:33 <anteaya> s/instructed/instructing
19:31:30 <jeblair> #topic  Open discussion
19:31:52 <jeblair> mordred: have you collapsed yet, or do you want to mention your work on manage-projects?
19:32:30 <clarkb> wait I have a thing
19:32:43 <SergeyLukjanov> fwiw, we hope to finish renaming of savanna to sahara this week, /me will have a bunch of clenups in random repos
19:32:45 <clarkb> https://review.openstack.org/#/c/51425/ should make our use of pip in puppet much much better
19:32:51 <clarkb> #link https://review.openstack.org/#/c/51425/
19:32:55 <clarkb> but it keeps running into rebase hell
19:33:12 <clarkb> I think the only way that gets merged is if we stop approving other puppet changes. Basically soft freeze before getting that in
19:33:22 <clarkb> btu I also think getting puppetboard working first would help a lot
19:33:33 <clarkb> so maybe we should bring this back up when we have puppetboard?
19:33:34 <pleia2> sounds good, I'm going through pip hell in fedora so maybe this will fix that
19:33:42 <anteaya> clarkb: so you are asking for a soft freeze after puppetboard is up, to get the pip patch in?
19:33:48 <pleia2> good == getting it in now
19:33:48 <fungi> clarkb: yes, after puppetboard is running
19:33:49 <clarkb> anteaya: yes
19:33:50 <jeblair> clarkb: yeah, let's do puppetboard first
19:33:53 <anteaya> soft freeze on puppet changes
19:33:54 <pleia2> aw :)
19:34:07 <clarkb> ok sounds good
19:34:14 <jeblair> so, puppetboard!  is that ready to go?
19:34:28 <nibalizer> uh the review is stll alive
19:34:32 <pleia2> I saw a review the other day
19:34:35 <jeblair> #link https://review.openstack.org/#/c/77928/
19:34:48 <nibalizer> jeblair: last thing i saw was you mentioning that it doesn't pip -U
19:34:56 <nibalizer> but then you +1'd so im not sure if thats a blocker or not
19:35:22 <SergeyLukjanov> folks, I'm planning to update my outdated CR that extracts some jobs to pypi-release template (https://review.openstack.org/#/c/76322/2/modules/openstack_project/files/zuul/layout.yaml) what do you think about it?
19:35:24 <jeblair> nibalizer: it's not, because keeping a system up to date with pip is nearly impossible anyway...
19:36:13 <jeblair> nibalizer: pip -U would be nice, but we've decided that running from packages is the best way to go
19:36:29 <fungi> SergeyLukjanov: i'm personally very much in favor of layout.yaml condensing with more templates
19:36:53 <jeblair> nibalizer: so anyway, i think the change is ok as is; then we should probably look into seeing if we can get pip -U in there, and eventually move to packages (farther in the future)
19:37:06 <SergeyLukjanov> fungi, I hope that someday we could have template "integrated-project"
19:37:08 <jeblair> SergeyLukjanov: ++
19:37:10 <nibalizer> jeblair: okay, so if someone with root wants to try some dry runs on puppetdb i think that intel will inform if that change can land right now
19:37:26 <SergeyLukjanov> fungi, jeblair, ok, thx, will do it today/tomorrow
19:37:31 <nibalizer> jeblair: i could add a weekly exec that activates the virtualenv and pip -U's
19:37:33 <mordred> jeblair: mostly collapsed - biggest thing to mention is the change that tries puppet via ssh
19:37:55 <fungi> jeblair: to your question yesterday about whether i'd tested the puppetboard change, now i remember i commented on ps4 saying i would test it but then mordred piped up in channel and said he was testing puppet changes anyway and would try it out
19:38:05 <clarkb> mordred: about that, why not salt? I think I missed that
19:38:15 <nibalizer> i'm also interested in feedback on https://review.openstack.org/#/c/71739/ because moving that forward should help a number of things
19:38:33 <jeblair> my changes to check and manage irc perms are about to merge, i think.  once they do, we should make sure that any new channels defined in any config files also exist in the file that the access bot uses
19:38:45 <jeblair> that's going to be a manual review criteria check for a while, until we can automate that
19:38:54 <mordred> clarkb: because the ssh change is pretty easy to grok and is not very complex and gets us to sequencing quickly
19:38:54 <clarkb> jeblair: noted
19:39:07 <clarkb> mordred: so does salt if you just do salt cmd 'review*' do thing
19:39:40 <clarkb> wasn't sure if ssh solved a particular problem that salt didn't
19:40:46 <jeblair> either way, we're talking about opening root access to all hosts; we probably want to restrict the commands that can be run
19:40:48 <mordred> clarkb: two things - one is expediency - which is that I could write that patch without going and reading a bunch of salt documentation
19:41:31 <mordred> jeblair: I was thinking about restricting ip perhaps? root on the puppetmaster automatically gets you root on the other boxes anyway because of puppet, no?
19:41:58 <nibalizer> mordred: thats correct
19:42:00 <jeblair> mordred: er, you want to only be able to ssh to hosts from the puppetmaster?
19:42:07 <mordred> as root
19:42:35 <jeblair> mordred: ssh-based ip restriction?
19:42:53 <mordred> yah
19:44:17 <jeblair> mordred: i'm not sure that gets us much; i don't think it's stronger than an rsa key...
19:44:39 <mordred> probably not, now you mention
19:44:54 <jeblair> mordred: i was getting at the fact that it would be nice if puppetmaster could do nothing other than run puppet on the other nodes
19:45:01 <fungi> i'm looking to see whether authorized_keys can associate source ip address with a specific public key
19:45:10 <fungi> vague recollection that was a feature
19:45:14 <jeblair> mordred: sure, it can have puppet do anything
19:45:53 <lifeless> ttx: I may be awol today, C is home sick from kindy
19:46:25 <jeblair> mordred: but it's at least a bit more auditable, if we get to that point in the future
19:46:53 <mordred> jeblair: nod. I'll think on that and come back with ideas
19:47:05 <ttx> lifeless: OK
19:47:15 <fungi> from="pattern-list" Specifies that in addition to public key authentication, either the canonical name of the remote host or its IP address must be present in the comma-separated list of patterns.
19:47:36 <jeblair> mordred: should be able to add a restricted command to the authorized_keys file
19:47:43 <fungi> so if we want to lock it down by a combination of ssh public key and source ip address that way, it's possible to do in the authorized_keys list
19:47:44 <jeblair> fungi: and i guess if it's that easy, we could add that too
19:47:56 <jeblair> probably wouldn't hurt
19:47:56 <fungi> jeblair: and we can limit commands there too, yes
19:48:13 <fungi> man 8 sshd /AUTHORIZED_KEYS FILE FORMAT
19:48:32 <fungi> there's a lot of space juice in there
19:48:49 <nibalizer> fungi: yes you can do that
19:48:59 <nibalizer> of course with ip/dns poisioning, it might not mean much
19:49:16 <jeblair> mordred: so this was pretty much to be our first step toward using salt to do complex orchestration across our hosts
19:49:33 <jeblair> mordred: if you're ditching that, should we just remove salt everywhere?
19:50:06 <fungi> nibalizer: right, more useful if we wanted to restrict a certain key to only be usable from a particular address, but i agree it doesn't buy much over just only installing the private key where you need it
19:50:24 <anteaya> part of my trip to thailand was to reset to focus on salt for Juno, from focusing on neutron for icehouse
19:50:47 <anteaya> so I have work to do to get some code up, but that was the direction I wanted to take for juno
19:50:53 <anteaya> unless I shouldn't
19:51:03 * anteaya considers heading back to thailand
19:51:56 <jeblair> anteaya: yeah, i'm a little sad because i think there's a lot we could do with salt.  but mordred volunteered for this implementation; i'm not sure what else we would use it for if we don't even want to use it for this...
19:51:59 <mordred> hrm. not sure. I think we could follow this with a patch that replaces the ssh commands with salt commands if we wanted
19:52:31 <fungi> one of the alternative fixes for the manage-projects race was to parse the replication config and loop retrying to clone each empty project from all the replication targets until they all exist before firing the replication command
19:52:41 <mordred> which might be easier to grok as a stepping stone - getting actually started with salt actually doing anything has seemed to be a bit of a roadblock
19:52:50 <anteaya> it is just that I need some prep work before I could offer anything, meaning we would have to limp along with manual manage-projects for a bit longer
19:53:35 <mordred> fungi: I'm not a fan of that personally - I do think we need ordered orchestration here and there are several ways to accomplish that
19:53:39 <anteaya> yes, I have been part of that roadblock, I kept getting distracted
19:53:48 <jeblair> fungi: i'll not that doesn't require a substatial change to our security posture
19:53:53 <jeblair> note
19:54:13 <jeblair> fungi: but it would have to check that before doing _anything_ with gerrit, not just the replication command
19:54:31 <jeblair> fungi: (since manage-projects actions in gerrit themselves could trigger replication actions)
19:54:36 <fungi> mordred: i agree we need some ordered orchestration overall, but if we need a quick fix to manage-projects while we roll out more robust orchestration, that seems preferable to rolling out scripted orchestration
19:54:46 <mordred> why?
19:54:53 <mordred> the poll seems like a horrible hack
19:55:03 <jesusaurus> can i get a link to the ssh patch? i'd be happy to help with salt stuff, but it's not clear to me what you guys want
19:55:05 <mordred> even worse than the ssh thing
19:55:27 <jeblair> s/horrible//: the thing has a dependency, it just checks to see if that dependency is satisfied...
19:55:39 <fungi> mordred: if you think the ssh thing will also be likely to only stick around until we can implement some more robust orchestration, then i'm fine with it
19:55:43 <mordred> but it's trying to do it multi-node
19:56:24 <mordred> sorry - my network is WAY too laggy to particpate well in an actual sensible discussion on this - I feel I'm doing a terrible job
19:56:30 <mordred> and coming across curt
19:56:31 <zaro> here is the etherpad of remaining gerrit upgrade changes:  https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes
19:56:33 <mordred> because of network
19:56:38 <fungi> just that salt, ansible, et al started out as projects to run a bunch of commands over ssh. they likely contain a lot of learned lessons
19:56:55 <anteaya> jesusaurus: https://review.openstack.org/#/c/80976/
19:57:01 <mordred> well, ansible did. salt does zeromq
19:57:28 <clarkb> zaro: you should use the #link directive with that so it shows up in meeting summaries
19:57:28 <fungi> fair. s/ssh/the network/
19:57:35 <jeblair> mordred: don't worry about it; i think you have some feedback, and we can regroup when you're in a better place...
19:57:45 <mordred> and I agree- but there is also apparently enough of a learning curve that we'v been stymied by this seemingly simple task for over a yaer
19:58:12 <jeblair> mordred: no one has worked on it until now
19:58:18 <mordred> I know
19:58:21 <clarkb> its only become necessary since the git farm went up
19:58:33 <fungi> mordred: i definitely agree that multi-dependency-oriented orchestration of tasks across separate systems in salt seems to be implemented in a way i've so far been unable to completely understand
19:58:33 <jesusaurus> why does the git farm make this necessary?
19:58:37 <mordred> every time I think about it I get stymied by learning curve
19:58:47 <clarkb> jesusaurus: because now we have cross node dependencies
19:58:50 <mordred> jesusaurus: because we ahve to add replicas before we add projects
19:58:56 <clarkb> jesusaurus: other than firewall rules which are fairly static we haven't had that much
19:59:03 <jesusaurus> gotcha
19:59:20 <jeblair> clarkb: and firewall rules work themselves out eventually
19:59:28 <mordred> jeblair: ++
19:59:33 <fungi> this particular issue would be moot if gerrit simply created the remote repositories via ssh if they didn't exist, before pushing commits into them
19:59:44 <jeblair> anyway, time's up; thanks everyone!
19:59:50 <mordred> woot
19:59:52 <jeblair> #endmeeting