19:02:11 <jeblair> #startmeeting infra 19:02:12 <openstack> Meeting started Tue Mar 18 19:02:11 2014 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:16 <openstack> The meeting name has been set to 'infra' 19:02:19 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 19:02:23 <jeblair> agenda ^ 19:02:27 <jeblair> #link http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-03-11-19.01.html 19:02:29 <jeblair> last meeting ^ 19:02:37 <fungi> yo 19:02:50 <anteaya> o/ 19:02:52 <jeblair> #topic Actions from last meeting 19:03:05 <jeblair> oh that was easy 19:03:09 <nibalizer> o/ 19:03:24 <jeblair> #topic Upgrade gerrit (zaro) 19:04:04 <jeblair> zaro: do you have the etherpad handy for this? 19:04:08 <zaro> changes waiting to be reviewed. 19:04:15 <zaro> ohh. let me find it. 19:04:31 <jeblair> zaro: i'd like to check in on what changes we need to review now, and which ones we should be planning on merging when we actually do the upgrade 19:05:20 <zaro> https://etherpad.openstack.org/p/gerrit-2.8-upgrade 19:05:25 <jeblair> #link https://etherpad.openstack.org/p/gerrit-2.8-upgrade 19:05:53 <zaro> here is the list 19:06:01 <zaro> #link https://review.openstack.org/#/c/79107 19:06:19 <zaro> #link https://review.openstack.org/#/c/70818/ 19:06:32 <zaro> #link https://review.openstack.org/#/c/60893/ 19:06:45 <zaro> #link https://review.openstack.org/#/c/60080/ 19:06:59 <zaro> #link https://review.openstack.org/#/c/69768/ 19:07:15 <zaro> #link https://review.openstack.org/#/c/70014/ 19:07:31 <zaro> #link https://review.openstack.org/#/c/70864/ 19:07:36 <jeblair> zaro: this week, can you update the etherpad and put those in 3 sections? 19:07:49 <jeblair> zaro: the changes we need to merge now, during the upgrade, and after the upgrade? 19:08:18 <zaro> sure. but that is all the remaining. 19:08:27 <jeblair> zaro: because it seems like some of those may not be safe to merge right now, right? 19:08:42 <zaro> maybe 2 are not, but rest can. 19:09:03 <zaro> i'll put it on etherpad and send out to irc. 19:09:14 <jeblair> zaro: coool thanks. 19:09:53 <zaro> ohh question. 19:10:11 <zaro> should we upgrade during openstack 'off' week? 19:10:24 <clarkb> I won't be around that week 19:10:48 <jeblair> #link https://wiki.openstack.org/wiki/Icehouse_Release_Schedule 19:10:53 <zaro> ohh yeah. i think clarkb was actually suggesting week before if we can gett all changes approved. 19:10:56 <clarkb> I was thinking week before might be slightly better as only fungi will be out that week iirc 19:11:02 <anteaya> the off week is designed for infra to be able to take a week off 19:11:08 <clarkb> anteaya: exactly :) 19:11:09 <anteaya> at least that was the intend I heard 19:11:16 <jeblair> clarkb: the week of 24? 19:11:30 <clarkb> jeblair: ya 19:11:49 * fungi will be missing in action for the two weeks prior to the off week, and then will be around for the off week, bizarrely 19:11:52 <clarkb> I get on a plane on the 26th (don't let me stop anyone from doing it during the off week fi you want to do it then) 19:11:54 <jeblair> anteaya: i don't think that was the intent -- i think the intent was that everyone should take a week off. it's more about release schedule timing than giving infra a break. 19:12:02 <jeblair> fungi: i will be around during the off week 19:12:10 <clarkb> jeblair: anteaya right it was an everyone off week including infra 19:12:17 <anteaya> oh I thought ttx thought infra needed to be able to take a break 19:12:46 <anteaya> and that a week needed to be scheduled since infra, release management and qa seem to work when everyone else is off 19:12:58 <fungi> i think ttx just needed to be able to go skiing ;) 19:13:03 <jeblair> anteaya: no, it's because the time between summits was too short for another milestone and too long for one less 19:13:10 <anteaya> oh okay 19:13:22 <jeblair> anteaya: and the time between the release and the summit is unstructured 19:13:42 <jeblair> anteaya: so having that time be too long means people are working on things that may not have been discussed or approved 19:13:44 * zaro is not off @ off week. 19:13:46 <jeblair> so we want to minimize it 19:14:37 <jeblair> zaro: let's see if a few more people will be around; if so, i like the idea of doing it during the off week 19:14:59 <pleia2> I'm around too 19:15:24 <clarkb> huh I figured everyone would be taking advantage of the off week 19:15:26 <fungi> i'm totally up for that 19:15:33 <fungi> people come back from break to a new gerrit ;) 19:15:47 <pleia2> clarkb: I still have some of my humanity, I don't plan my *whole* life around the openstack release cycle :) 19:15:48 <mordred> oh. 19:15:50 <mordred> o/ 19:15:53 <mordred> I'm actually awake still 19:16:03 <jeblair> mordred: have plans for the off week? https://wiki.openstack.org/wiki/Icehouse_Release_Schedule 19:16:10 <jeblair> mordred: we're thinking of doing the gerrit upgrade then 19:16:11 <fungi> pleia2: we'll fix that soon, i promise 19:16:13 <clarkb> pleia2: it actually worked out great for me so win for me I guess 19:16:16 <pleia2> fungi: haha 19:16:33 <mordred> jeblair: works for me 19:16:34 <pleia2> I'm only gone for the weekend (conference) 19:17:01 <jeblair> clarkb, fungi: i'm going to be tooling around NC the week between the off week and summit (and then road tripping to the summit) 19:17:15 <jeblair> clarkb: so that's great that you'll be around to fix any problems that crop up then! ;) 19:17:25 <anteaya> I'm around that week 19:17:35 <fungi> jeblair: i'm around that week too, so should be fine 19:17:36 <clarkb> jeblair: ha 19:18:00 <fungi> jeblair: we'll be running a whole new fork of gerrit when you get back ;) 19:18:54 <jeblair> fungi: and after my next vacation, vinz? :) 19:19:02 <fungi> likely 19:19:11 <pleia2> :) 19:19:13 <clarkb> no no first pygit needs to be fixed :) 19:19:37 <jeblair> ok, so let's tentatively say we'll upgrade during the off week.... 19:19:47 <clarkb> wfm 19:19:55 <jeblair> zaro: do you want to send a note to the ml suggesting that and see if anyone screams? 19:20:05 <zaro> will do. 19:20:23 <jeblair> if no one does, we can start doing more formal announcements closer to then 19:20:28 <zaro> ohh is that openstack-infra ml or opestack ml? 19:20:35 <clarkb> openstack-dev 19:20:39 <zaro> ok 19:20:42 <clarkb> is probably the best place to get a tempurature 19:21:20 <jeblair> #topic determining atc for the purposes of ptl and tc elections (anteaya) 19:21:45 <fungi> ooh! 19:21:46 <anteaya> o/ 19:22:00 <mordred> look. it's an anteaya 19:22:05 <jeblair> anteaya: welcome back! 19:22:11 <fungi> anteaya: we run a script to generate that currently, from the programs list in the gobernance repo 19:22:14 <jeblair> in the agenda, anteaya says " does using projects.yaml from the governance repo get us an accurate list? " 19:22:16 <fungi> governance 19:22:19 <anteaya> so the governance repo has a handy projects.yaml: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml 19:22:21 <jeblair> #link http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml 19:22:28 <anteaya> jeblair: thanks 19:22:53 <fungi> anteaya: that is the list we use. are you saying you believe it to be inaccurate? 19:23:02 <anteaya> so what would happen if that was used as a reference for atc list generation for voters 19:23:19 <anteaya> fungi: no, just wanting to ensure I am using it the correct way 19:23:25 <fungi> yep 19:23:27 <anteaya> for instance Barbican is on that list 19:23:41 <anteaya> but to my knowledge is not an integrated project 19:23:43 <fungi> when did barbican join a tc-approved program? 19:23:50 <fungi> right 19:23:53 <anteaya> that was my question 19:23:54 <clarkb> git log? 19:23:58 <mordred> it's incubated 19:24:05 <mordred> incubated counts for atc 19:24:13 <anteaya> when did it get incubated? 19:24:16 <fungi> that is the list official openstack programs use for determining electorate, and the foundation uses to determine who's an active technical contributor for an official project 19:24:18 <anteaya> I must have missed that 19:24:42 <fungi> and since it's in code review, anyone who spots an error can propose a fix and let the tc vote to approve the update 19:24:51 <jeblair> #link https://review.openstack.org/#/c/77647/ 19:24:55 <anteaya> great, so I will use it for electorate generation and evaluation 19:25:30 <anteaya> March 10 19:25:33 <anteaya> jeblair: thanks 19:25:44 <anteaya> okay, my question is answered, thank you 19:25:47 <jeblair> so, looks like it's time to start moving barbican to openstack/ 19:26:11 <fungi> sounds like it. maybe climate can settle on a new name asap and we can rename them all at the same shot? 19:26:26 <clarkb> wait more renaming? 19:26:29 <jeblair> fungi: i'll add both to the agenda 19:26:52 <clarkb> should we add a review thing to new project addtions that includes 'googled project name and function didn't see any relevant hits" 19:27:18 <zaro> do all incubated projects live in stackforge? 19:27:25 <jeblair> zaro: no, they live in openstack/ 19:27:45 <jeblair> zaro: much easier to move them when the move into incubation than when they graduate 19:28:04 <jeblair> also, they become official projects at that point, so it makes sense 19:28:05 <pleia2> pre-incubation they typically are in stackforge 19:28:21 <ttx> (late remark: the idea behind the off week is that 3 weeks is way too much time between release and summit, so we might as well try to encourage a vacation week in there) 19:28:22 <jeblair> pleia2: ++ and actually the tc has made that a requirement now 19:28:29 <pleia2> jeblair: ah, nice 19:28:36 <zaro> so what's the logical seperation between incubated and official? is does none exist? 19:28:38 <jeblair> ttx: we interpreted "vacation" as "upgrade gerrit" :) 19:28:48 <clarkb> except for me 19:28:53 <fungi> right, at least when they're still only incubated, there aren't usually cross-dependencies requiring coordinated changes to how we're testing any official projects during the rename 19:28:55 <anteaya> fungi: the script that runs on the governance/projects.yaml it uses the preferred email from gerrit, yes? 19:28:59 * clarkb is going to vacation in hawaii that week 19:29:01 <pleia2> zaro: repo organization-wise, none really 19:29:07 <anteaya> yay clarkb 19:29:11 <ttx> jeblair: whatever you like to do as vacation ! 19:29:18 <jeblair> zaro: incubated is a subset of official (integrated is also a subset of official) 19:29:42 <fungi> anteaya: yes, we run a database query locally on the gerrit server's backend and then feed that list of addresses and activity into the script 19:29:43 * ttx will be in Barcelona 19:29:45 <jeblair> zaro: incubated and integrated are disjoint subsets of official 19:30:21 <anteaya> fungi: thought so, just wanted to confirm since I will be instructed electorate to confirm the gerrit prefered address in election prep 19:30:33 <anteaya> s/instructed/instructing 19:31:30 <jeblair> #topic Open discussion 19:31:52 <jeblair> mordred: have you collapsed yet, or do you want to mention your work on manage-projects? 19:32:30 <clarkb> wait I have a thing 19:32:43 <SergeyLukjanov> fwiw, we hope to finish renaming of savanna to sahara this week, /me will have a bunch of clenups in random repos 19:32:45 <clarkb> https://review.openstack.org/#/c/51425/ should make our use of pip in puppet much much better 19:32:51 <clarkb> #link https://review.openstack.org/#/c/51425/ 19:32:55 <clarkb> but it keeps running into rebase hell 19:33:12 <clarkb> I think the only way that gets merged is if we stop approving other puppet changes. Basically soft freeze before getting that in 19:33:22 <clarkb> btu I also think getting puppetboard working first would help a lot 19:33:33 <clarkb> so maybe we should bring this back up when we have puppetboard? 19:33:34 <pleia2> sounds good, I'm going through pip hell in fedora so maybe this will fix that 19:33:42 <anteaya> clarkb: so you are asking for a soft freeze after puppetboard is up, to get the pip patch in? 19:33:48 <pleia2> good == getting it in now 19:33:48 <fungi> clarkb: yes, after puppetboard is running 19:33:49 <clarkb> anteaya: yes 19:33:50 <jeblair> clarkb: yeah, let's do puppetboard first 19:33:53 <anteaya> soft freeze on puppet changes 19:33:54 <pleia2> aw :) 19:34:07 <clarkb> ok sounds good 19:34:14 <jeblair> so, puppetboard! is that ready to go? 19:34:28 <nibalizer> uh the review is stll alive 19:34:32 <pleia2> I saw a review the other day 19:34:35 <jeblair> #link https://review.openstack.org/#/c/77928/ 19:34:48 <nibalizer> jeblair: last thing i saw was you mentioning that it doesn't pip -U 19:34:56 <nibalizer> but then you +1'd so im not sure if thats a blocker or not 19:35:22 <SergeyLukjanov> folks, I'm planning to update my outdated CR that extracts some jobs to pypi-release template (https://review.openstack.org/#/c/76322/2/modules/openstack_project/files/zuul/layout.yaml) what do you think about it? 19:35:24 <jeblair> nibalizer: it's not, because keeping a system up to date with pip is nearly impossible anyway... 19:36:13 <jeblair> nibalizer: pip -U would be nice, but we've decided that running from packages is the best way to go 19:36:29 <fungi> SergeyLukjanov: i'm personally very much in favor of layout.yaml condensing with more templates 19:36:53 <jeblair> nibalizer: so anyway, i think the change is ok as is; then we should probably look into seeing if we can get pip -U in there, and eventually move to packages (farther in the future) 19:37:06 <SergeyLukjanov> fungi, I hope that someday we could have template "integrated-project" 19:37:08 <jeblair> SergeyLukjanov: ++ 19:37:10 <nibalizer> jeblair: okay, so if someone with root wants to try some dry runs on puppetdb i think that intel will inform if that change can land right now 19:37:26 <SergeyLukjanov> fungi, jeblair, ok, thx, will do it today/tomorrow 19:37:31 <nibalizer> jeblair: i could add a weekly exec that activates the virtualenv and pip -U's 19:37:33 <mordred> jeblair: mostly collapsed - biggest thing to mention is the change that tries puppet via ssh 19:37:55 <fungi> jeblair: to your question yesterday about whether i'd tested the puppetboard change, now i remember i commented on ps4 saying i would test it but then mordred piped up in channel and said he was testing puppet changes anyway and would try it out 19:38:05 <clarkb> mordred: about that, why not salt? I think I missed that 19:38:15 <nibalizer> i'm also interested in feedback on https://review.openstack.org/#/c/71739/ because moving that forward should help a number of things 19:38:33 <jeblair> my changes to check and manage irc perms are about to merge, i think. once they do, we should make sure that any new channels defined in any config files also exist in the file that the access bot uses 19:38:45 <jeblair> that's going to be a manual review criteria check for a while, until we can automate that 19:38:54 <mordred> clarkb: because the ssh change is pretty easy to grok and is not very complex and gets us to sequencing quickly 19:38:54 <clarkb> jeblair: noted 19:39:07 <clarkb> mordred: so does salt if you just do salt cmd 'review*' do thing 19:39:40 <clarkb> wasn't sure if ssh solved a particular problem that salt didn't 19:40:46 <jeblair> either way, we're talking about opening root access to all hosts; we probably want to restrict the commands that can be run 19:40:48 <mordred> clarkb: two things - one is expediency - which is that I could write that patch without going and reading a bunch of salt documentation 19:41:31 <mordred> jeblair: I was thinking about restricting ip perhaps? root on the puppetmaster automatically gets you root on the other boxes anyway because of puppet, no? 19:41:58 <nibalizer> mordred: thats correct 19:42:00 <jeblair> mordred: er, you want to only be able to ssh to hosts from the puppetmaster? 19:42:07 <mordred> as root 19:42:35 <jeblair> mordred: ssh-based ip restriction? 19:42:53 <mordred> yah 19:44:17 <jeblair> mordred: i'm not sure that gets us much; i don't think it's stronger than an rsa key... 19:44:39 <mordred> probably not, now you mention 19:44:54 <jeblair> mordred: i was getting at the fact that it would be nice if puppetmaster could do nothing other than run puppet on the other nodes 19:45:01 <fungi> i'm looking to see whether authorized_keys can associate source ip address with a specific public key 19:45:10 <fungi> vague recollection that was a feature 19:45:14 <jeblair> mordred: sure, it can have puppet do anything 19:45:53 <lifeless> ttx: I may be awol today, C is home sick from kindy 19:46:25 <jeblair> mordred: but it's at least a bit more auditable, if we get to that point in the future 19:46:53 <mordred> jeblair: nod. I'll think on that and come back with ideas 19:47:05 <ttx> lifeless: OK 19:47:15 <fungi> from="pattern-list" Specifies that in addition to public key authentication, either the canonical name of the remote host or its IP address must be present in the comma-separated list of patterns. 19:47:36 <jeblair> mordred: should be able to add a restricted command to the authorized_keys file 19:47:43 <fungi> so if we want to lock it down by a combination of ssh public key and source ip address that way, it's possible to do in the authorized_keys list 19:47:44 <jeblair> fungi: and i guess if it's that easy, we could add that too 19:47:56 <jeblair> probably wouldn't hurt 19:47:56 <fungi> jeblair: and we can limit commands there too, yes 19:48:13 <fungi> man 8 sshd /AUTHORIZED_KEYS FILE FORMAT 19:48:32 <fungi> there's a lot of space juice in there 19:48:49 <nibalizer> fungi: yes you can do that 19:48:59 <nibalizer> of course with ip/dns poisioning, it might not mean much 19:49:16 <jeblair> mordred: so this was pretty much to be our first step toward using salt to do complex orchestration across our hosts 19:49:33 <jeblair> mordred: if you're ditching that, should we just remove salt everywhere? 19:50:06 <fungi> nibalizer: right, more useful if we wanted to restrict a certain key to only be usable from a particular address, but i agree it doesn't buy much over just only installing the private key where you need it 19:50:24 <anteaya> part of my trip to thailand was to reset to focus on salt for Juno, from focusing on neutron for icehouse 19:50:47 <anteaya> so I have work to do to get some code up, but that was the direction I wanted to take for juno 19:50:53 <anteaya> unless I shouldn't 19:51:03 * anteaya considers heading back to thailand 19:51:56 <jeblair> anteaya: yeah, i'm a little sad because i think there's a lot we could do with salt. but mordred volunteered for this implementation; i'm not sure what else we would use it for if we don't even want to use it for this... 19:51:59 <mordred> hrm. not sure. I think we could follow this with a patch that replaces the ssh commands with salt commands if we wanted 19:52:31 <fungi> one of the alternative fixes for the manage-projects race was to parse the replication config and loop retrying to clone each empty project from all the replication targets until they all exist before firing the replication command 19:52:41 <mordred> which might be easier to grok as a stepping stone - getting actually started with salt actually doing anything has seemed to be a bit of a roadblock 19:52:50 <anteaya> it is just that I need some prep work before I could offer anything, meaning we would have to limp along with manual manage-projects for a bit longer 19:53:35 <mordred> fungi: I'm not a fan of that personally - I do think we need ordered orchestration here and there are several ways to accomplish that 19:53:39 <anteaya> yes, I have been part of that roadblock, I kept getting distracted 19:53:48 <jeblair> fungi: i'll not that doesn't require a substatial change to our security posture 19:53:53 <jeblair> note 19:54:13 <jeblair> fungi: but it would have to check that before doing _anything_ with gerrit, not just the replication command 19:54:31 <jeblair> fungi: (since manage-projects actions in gerrit themselves could trigger replication actions) 19:54:36 <fungi> mordred: i agree we need some ordered orchestration overall, but if we need a quick fix to manage-projects while we roll out more robust orchestration, that seems preferable to rolling out scripted orchestration 19:54:46 <mordred> why? 19:54:53 <mordred> the poll seems like a horrible hack 19:55:03 <jesusaurus> can i get a link to the ssh patch? i'd be happy to help with salt stuff, but it's not clear to me what you guys want 19:55:05 <mordred> even worse than the ssh thing 19:55:27 <jeblair> s/horrible//: the thing has a dependency, it just checks to see if that dependency is satisfied... 19:55:39 <fungi> mordred: if you think the ssh thing will also be likely to only stick around until we can implement some more robust orchestration, then i'm fine with it 19:55:43 <mordred> but it's trying to do it multi-node 19:56:24 <mordred> sorry - my network is WAY too laggy to particpate well in an actual sensible discussion on this - I feel I'm doing a terrible job 19:56:30 <mordred> and coming across curt 19:56:31 <zaro> here is the etherpad of remaining gerrit upgrade changes: https://etherpad.openstack.org/p/remaining-gerrit-upgrade-changes 19:56:33 <mordred> because of network 19:56:38 <fungi> just that salt, ansible, et al started out as projects to run a bunch of commands over ssh. they likely contain a lot of learned lessons 19:56:55 <anteaya> jesusaurus: https://review.openstack.org/#/c/80976/ 19:57:01 <mordred> well, ansible did. salt does zeromq 19:57:28 <clarkb> zaro: you should use the #link directive with that so it shows up in meeting summaries 19:57:28 <fungi> fair. s/ssh/the network/ 19:57:35 <jeblair> mordred: don't worry about it; i think you have some feedback, and we can regroup when you're in a better place... 19:57:45 <mordred> and I agree- but there is also apparently enough of a learning curve that we'v been stymied by this seemingly simple task for over a yaer 19:58:12 <jeblair> mordred: no one has worked on it until now 19:58:18 <mordred> I know 19:58:21 <clarkb> its only become necessary since the git farm went up 19:58:33 <fungi> mordred: i definitely agree that multi-dependency-oriented orchestration of tasks across separate systems in salt seems to be implemented in a way i've so far been unable to completely understand 19:58:33 <jesusaurus> why does the git farm make this necessary? 19:58:37 <mordred> every time I think about it I get stymied by learning curve 19:58:47 <clarkb> jesusaurus: because now we have cross node dependencies 19:58:50 <mordred> jesusaurus: because we ahve to add replicas before we add projects 19:58:56 <clarkb> jesusaurus: other than firewall rules which are fairly static we haven't had that much 19:59:03 <jesusaurus> gotcha 19:59:20 <jeblair> clarkb: and firewall rules work themselves out eventually 19:59:28 <mordred> jeblair: ++ 19:59:33 <fungi> this particular issue would be moot if gerrit simply created the remote repositories via ssh if they didn't exist, before pushing commits into them 19:59:44 <jeblair> anyway, time's up; thanks everyone! 19:59:50 <mordred> woot 19:59:52 <jeblair> #endmeeting