19:01:02 #startmeeting infra 19:01:03 Meeting started Tue May 12 19:01:02 2015 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:06 The meeting name has been set to 'infra' 19:01:09 o/ 19:01:11 #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:11 #link previous meeting http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-05-05-19.03.html 19:01:11 o/ 19:01:12 o/ 19:01:29 #topic Announcements 19:01:33 #link design summit schedule http://libertydesignsummit.sched.org/type/design+summit/infrastructure#.VVJOMuSVtpg 19:01:54 that's what i pulled together out of the etherpad we were working on 19:02:03 o/ 19:02:13 it is, as always, subject to change, especially this week 19:02:35 o/ 19:02:44 you'll note that the "work sessions" have boring titles but actual descriptions of what they entail 19:02:59 o/ 19:03:07 yay for boring titles 19:03:18 the intent (this is an openstack-wide thing, not an infra decision) is to keep people from just showing up because the title looks interesting 19:03:35 so while we might expect a little bit of an audience (though it's still not encouraged) for the fishbowl sessions 19:03:58 the work sessions should be much less of a 'show' 19:04:16 o/ 19:04:20 and then the meetup has no fixed agenda or even a description attached 19:04:54 so while we still have things that we think we're going to hack on going in, but we can decide on our agenda on that day 19:05:14 any questions or comments? 19:05:18 there's also a translations session on *monday* that some of us should try to attend (I'm running it with Daisy), details coming together here https://etherpad.openstack.org/p/Vancouver-I18n-WG-session I expect we'll want use some time on Friday for this work too since all the right people will be together in person (and time zone!), like we did in Paris 19:05:56 pleia2: what's the time slot for that? 19:06:11 jeblair: tentatively 2:50pm - 3:30pm 19:06:37 the etherpad will be updated if that changes, and I'll try to let people know 19:06:50 fyi - I will be in an ansible session monday afternoon- anyone who wants to talk about the new ansible modules is welcome to come 19:07:10 this is part of the "associated projects get space near the summit" program 19:07:13 presentation or discussion? 19:07:19 discussion 19:07:26 cool, I would like to attend 19:08:14 #topic Actions from last meeting 19:08:21 fungi check our cinder quota in rax-dfw 19:08:27 not sure if fungi is back yet 19:08:31 #action fungi check our cinder quota in rax-dfw 19:08:37 #action jeblair make the priority effort gerrit topics more visible to reduce the need to shout for reviews in this meeting 19:08:53 i am almost there ^ 19:09:00 i think i'll be able to get to that this week 19:09:01 mordred: is there a link to a schedule thing for that? 19:09:38 tchaypo: nope - I'll try to get people info 19:09:47 i am back now 19:09:54 sorry about that 19:10:02 wb fungi 19:10:11 and no, i did not get around to checking our quota ;) 19:10:21 excellent, action item already action itemed! 19:10:35 your precognitive skills are beyond compare 19:10:37 mordred, Ya, plan to attend the ansible stuff too. See that the vibe is like 19:10:54 #topic Schedule next project renames 19:11:02 i don't want to do this now, does anyone else? :) 19:11:07 ha ha ha 19:11:12 post summit 19:11:39 nope 19:11:47 yep, let's do this next post summit 19:11:49 #topic Priority Efforts (Downstream Puppet) 19:11:49 i have to catch a flight very early saturday, so er, friday afternoon would even be a bit of a stretch for me 19:12:03 ya and everyone is busy enough with summit prep 19:12:10 nibalizer: you pinged? 19:12:30 yes so https://review.openstack.org/#/c/178887/ is part of a plan cooked up in -infra 19:12:38 as a way to accoplish the linked spec 19:12:49 so sorry for anyone who is administratively -2'd for that 19:12:59 the dependent change (causing the admin -2s) has merged I think at this point though 19:13:04 erp 19:13:23 erp this is the review i meant https://review.openstack.org/#/c/180783/ 19:13:46 #link https://review.openstack.org/180783 19:13:55 jeblair: point is, I think the only refactors I needed to do touching o_p::template and o_p::server are done 19:13:57 oh this is the template reorg? 19:14:03 so those admin -2's are no longer needed 19:14:05 jeblair: yes 19:14:19 nibalizer: excellent, i will remove them asap, thanks 19:14:31 and for those following along the -2s were on other changes to avoid conflicts 19:14:33 this spec http://git.openstack.org/cgit/openstack-infra/infra-specs/tree/specs/server_base_template_refactor.rst 19:14:39 clarkb: thanks 19:14:48 jeblair: well thats what I think... do you concur that we dont' need the -2's any more? 19:15:22 better link: http://specs.openstack.org/openstack-infra/infra-specs/specs/server_base_template_refactor.html 19:15:45 nibalizer: they were mostly to make your life easier and avoid lots of rebases, so i'm good if you're good :) 19:16:01 okay, good :) 19:16:12 then they can be removed, thanks for helping with that 19:16:22 np 19:16:25 and thats all I had from a 'downstream-puppet' place 19:16:43 any other priority effort topics? 19:17:09 oh I have one 19:17:13 no blockers to report for the stuff i'm working on anyway 19:17:30 jhesketh has written a devstack plugin for os-loganalyze, which will allow us to test that with real swift 19:17:42 just pointing it out because its a thing and not something we have done before 19:17:57 that's kinda cool :) 19:18:00 great idea 19:18:00 but its possible nodepool may want to do something similar 19:18:06 well, almost - we are doing functional testing with shade against devstack too 19:18:09 askbot at my side 19:18:11 however, I do agree it's super cool 19:18:14 as the next step beyond the "thsi is how you nodepool + devstack" 19:18:21 mordred: you aren't using a plugin are you? 19:18:24 clarkb: no need to 19:18:35 oh right since shade isn't a service 19:18:54 make cloud, point shade at cloud' 19:19:07 I fully support the use of devstack nodes to test infra uses of openstacks 19:19:12 #topic Priority Efforts (Swift logs) 19:19:33 sorry, late on the topic switch there :) 19:19:58 * mordred hands jeblair a sad looking bowl of alfalfa 19:20:12 * jeblair ruminates 19:20:32 mrmartin: did you want to talk about askbot? 19:20:51 yeap 19:20:52 just a short comment here 19:20:54 19:20 < jeblair> mrmartin: did you want to talk about askbot? 19:20:57 #topic Priority Efforts (Askbot migration) 19:20:58 is swift log something we can setup for review-dev as well? 19:21:07 ok, I'll be quick 19:21:25 zaro: it should be completely independent of review-dev 19:21:33 zaro: it is for zuul jobs 19:21:40 so we have an issue with actual ask.o.o google auth, because it is deprecated, and need to change to g+ 19:21:59 mrmartin: specifically openid right? 19:22:00 clarkb: we can talk about it later since topic has turned. 19:22:11 clarkb: yeap 19:22:34 so we the askbot-devel (which is a remote github repo independent from us) contains all of the patches now 19:23:21 but as we discussed previously, we need to consume askbot-devel from this github repo directly, which requires a lot of changes in puppet-askbot module 19:23:52 mrmartin: the big difference being pypi release vs source install? 19:23:53 and to make the story nice, I was running into different issues that was related to askbot release management and the missing Q/A 19:24:00 clarkb: yes 19:25:03 mrmartin: would they be so kind as to make a release? 19:25:14 sounds like our askbot dev/sandbox server is going to turn into upstream askbot qa 19:25:46 fungi: it won't be the first project that's the case for 19:25:46 fungi: it won't be a first for openstack :/ 19:25:49 wow 19:25:51 hah 19:26:13 jeblair: last time we agreed to use master branch and feature / development branch to separate the testing and release branches 19:26:21 so the actual puppet-askbot covers that 19:26:26 oh ok 19:26:28 mrmartin: well, that was for the askbot-theme repo, right? 19:26:40 (or whatever it's called) 19:26:48 yeah, askbot-theme was the easy part 19:26:55 but we are doing the same for askbot-devel (the external one) 19:27:11 http://git.openstack.org/cgit/openstack-infra/puppet-askbot/tree/manifests/init.pp#n59 19:27:12 so we don't need to push the version numbers of pip packages for every release 19:27:27 right now it only installs askbot via pip 19:27:32 until the package maintainer keeps that policy 19:27:33 yes 19:27:59 so it is a major change, and it was the reason the I commented out all the askbot related resource in the current ask.o.o manifests 19:28:15 and it is a great momemnt to change from precise to trusty 19:29:02 i'm starting to lose the narrative thread here... is there a proposal we should be discussing? 19:29:04 so first we can deploy the askbot-staging.o.o based on new manifests, and if it works well, it is easy to upgrade the ask.o.o site 19:29:15 i'm a little worried about trying to arrange another askbot production server migration in the next few days before so many of us pack up and ship out to vancouver 19:29:24 but staging seems doable 19:29:36 I'm sure we cannot to do the prod server migration before the summit 19:29:40 its also worth pointing out that issues like this are why we gave up on askbot in the past 19:30:08 its great that we are motivated to help fix things so ++ on making stuff go 19:30:15 so the question here, fungi, do you have an idea how to deliver the askbot patches with google-plus auth changes within a reasonable time? 19:31:10 I vote to keep the actual workflow, and deploy askbot-staging before the summit if we can, and do the prod migration after summit 19:31:14 i guess it depends on how we define reasonable 19:31:21 :) 19:31:28 that seems like a timeframe i can help with 19:31:35 okay, so let me see if i can summarize: needed patches are in the upstream askbot repo. they refuse to make a release. you think the current state of the askbot upstream repo is not production ready. you want to deploy from git on the staging site. then you want to later deploy from git (what version? some tested hash?) on the production site. 19:31:44 i wanted to get back to reviewing your outstanding puppet patches for teh dev server today 19:32:10 * nibalizer also has the askbot puppet patches next in the stack of things to review 19:32:41 jeblair: we deploy the master branch from askbot-devel repo, and they'll deliver dev patches into a separate branch 19:33:17 so we can support rolling releases this way, and we don't need to handle the specific version numbers or tags for every new production grade release 19:33:18 so basically "master" is intended to be their stable branch upstream, sounds like 19:33:23 yes 19:33:45 ok, fwiw, we can deploy the latest pip release too 19:34:05 so deploying from git is not required in order to have auto rolling upgrades 19:34:28 deploying from git is fine if that's what we want to do, just wanted to make sure we weren't doing it solely because we didn't want to bump version numbers 19:34:40 ok, but with this suggesting we'll deploy from git for staging and pip for production? 19:34:52 I support to deploy from a single source 19:35:21 ok. 19:35:22 jeblair: keep in mind they're not "our" version numbers or "our" repos or "our" packages on pypi 19:35:30 fungi: i understand 19:35:36 I know, but the same is true for pypi 19:35:56 we should deploy from whatever is appropriate for the project 19:35:58 so it sounds like we want to deploy from git to get fixes which are not in the release packages 19:35:59 so it will work until the app maintainer keeps the policy of master / feature branch commit 19:36:13 i was specifically responding to the idea that we should use git because mrmartin did not want to bump version numbers 19:36:43 if "upstream git master" is actually "the maintained stable distribution of askbot", that's fine. i have no idea. :) 19:36:51 oh, got it. i interpreted that as we wanted to deploy from git because evgeny doesn't want to bump version numbers 19:36:52 to clarify I don't have any problem with version numbers, I have a problem with a different method of deployment of staging and prod servers 19:37:59 mrmartin: okay, so you want to deploy both sites from git. do you want both to track git master? 19:38:01 jeblair: yes, so it's not an openstack project, and we just have a little influence what happens there 19:38:10 got it. so if he's at least tagging versions in git and those are versions we want to use, it's easy to use one mechanism to deploy from a tag name or a branch or a specific sha 19:38:26 no, the staging will track git / feature /development branch, and prod will track the master 19:39:07 but it is up to us, because the Vcsrepo gives us the freedom to select between tags or master branch 19:39:26 okay. i am surprised that the outcome of a discussion around whether askbot upstream can be relied upon to make reliable stable releases is "production should continuously track upstream master", but i will take your word for it :) 19:39:47 that seems fine to me. it's not dissimilar to how we're consuming etherpad 19:39:51 I've spent the last two weeks to remove bugs from the new release and reverse engineer why some non-documented config setting was changed 19:39:55 and yeah, if/when it breaks, we can always pin a tag or commit 19:39:59 ++ 19:40:01 yes 19:40:08 i mean, it's certainly not an ideal development model, but we're not the upstream devs 19:40:26 it is not an ideal development model, I agree, but that's we have actually 19:40:53 #agreed switch puppet-askbot to deploy from git 19:40:56 yeah? ^ 19:41:05 in an ideal world I like to move-in the entire project under our gating / code review system, or build-up a proper governance model for the project 19:41:07 yeap 19:41:20 also having it deployed from vcsrepo in puppet makes it marginally easier for us to switch to a fork of the project if we absolutely have to 19:41:24 #agreed switch askbot-staging to CD from upstream git development branch 19:41:41 #agreed switch askbot production to CD from upstream git master branch (some time after openstack summit) 19:41:50 i have all that right? 19:41:58 ok, great, thanks 19:42:00 that looks correct to me 19:42:33 mrmartin: thanks, i think this discussion was really helpful 19:42:44 anything else on askbot? 19:42:53 and deploy the askbot-staging if we can before the summit :) 19:43:35 right; time permitting :) 19:44:07 #topic Priority Efforts (Upgrading Gerrit) 19:44:18 so, i reckon we ought to chat about this 19:44:26 as fungi said: "this did not work as planned" 19:44:50 * fungi sighs 19:44:59 from my pov, we're safely back on 2.8, and we are not under high pressure to upgrade immediately 19:45:03 no response from mailing list yet 19:45:03 sufferring is defined as reality not meeting pre-conceived expectations 19:45:20 jeblair: ++ 19:45:28 jeblair: yup, stevebaker ran into some trouble though, was hoping stevebaker would respond to the upgrade/downgrade thread with details but I didn't see that 19:45:42 so i think the next steps are to reproduce the problem out of production, and when that happens, we can develop/confirm a fix, then talk about trying it again. 19:45:51 agree 19:45:52 any aftermath() from downgrade that i don't know of? 19:45:54 one of the nova devs mentioned to me last night that the impact was "barely noticed" so maybe we do a much better job of shielding our dev community from this sort of havoc than we think we do at least 19:46:15 clarkb: er, do we have _any_ details on that trouble? 19:46:17 zaro: stevebaker does not have a working web client 19:46:35 jeblair: from what I could gather late last night when logged in stevebaker's web client throws and exception and does nothing 19:46:37 I had some ideas on some ways in which we might want to structure review-dev to accomplish this - I will write them up in a spec 19:47:04 jeblair: he thought it may be because he had made settings changes to his account on the 2.10 side but looking at the accounts table where that appeared to be stored I didn't see anything off 19:47:24 fungi: yeah, but i know that a significant number of old changes were causing complete failures, and that in turn was taking out dashboards, queries, etc; so i think our state on 2.10 was unsustainable in the long term 19:47:36 mordred: thanks 19:47:56 i'm glad that people were able to get some work done while we futzed with it though :) 19:48:07 mordred: clarkb and i discussed putting copying data from review to review-dev. 19:48:08 jeblair: no question there, just saying i feel a little bit better about the actual perceived impact 19:48:36 zaro: yup. I agree with that 19:49:12 jeblair: maybe someone else can look at that row in the accounts table and tell me if they notice anything off? 19:49:23 i should clarify, we planned to copy over all the data with passwords removed. 19:49:26 I basically compared my account to stevebaker's 19:50:31 oh, as far as dumping production data, i did successfully script stripping out the http passwords from a mysqldump at one point. i can dig that back up 19:50:48 if it's of potential help 19:51:04 fungi: I think that would help yes 19:51:13 also might need to simulate activity on review-dev somehow. 19:51:20 i think a quick spec for how to reconfigure review-dev would be good -- there are things we can do in the current setup that we may not be able to with a production copy, and vice versa 19:51:35 so would be good to get all on the same page there 19:51:45 ++ 19:52:13 and sounds like that's the next step for this effort 19:52:15 generating a lot of load is probably not hard. simulating the nature of our production load is probably not going to be feasible but hopefully also unnecessary to reproduce the error 19:52:52 should we hold off putting review data on review-dev until spec is reviewed? 19:52:55 anyway, thanks to everyone that worked on the upgrade, and sorry it didn't work out 19:53:11 oh maybe we can diff stevebaker's account row between previous 2.8 state and current 19:53:25 mordred: ^ maybe you can point me at where the old copy is 19:53:31 zaro: if we can, or at least let's not permanently delete anything from review-dev yet 19:53:41 zaro: i think we probably should hold off, yes, we want to make sure we don't cause any information leakage or make a nuisance out of it 19:53:59 clarkb: it's in root's home dir 19:54:03 mordred: thanks 19:54:07 roger that 19:54:11 #topic Open discussion 19:54:38 jeblair: do you plan to create skeletons for all etherpadds or shall moderators create each ? 19:54:47 (infra track session etherpads) 19:54:49 probably goes without saying, but just in case, i don't think we'll have an infra meeting next week :) 19:54:58 :) 19:55:01 https://review.openstack.org/#/c/178887/ hasn't converged so everyone go weigh in on that 19:55:13 it sounds like most of the momentum for puppet functional testing is behind beaker. Thats fine but one of the first things we will need to solve is how to configure module repos such that puppet tests the correct stuff in the gate and on a developers laptop 19:55:20 clarkb: so what i meant was that logs on review-dev aren't accessible from gerrit. would like to see that work. 19:55:31 ttx: oh good question; moderators i think 19:55:32 quick note from me, grafyaml is shaping up nice. Have a review up for it to pull under -infra: https://review.openstack.org/#/c/182045/ 19:55:40 ok, will create mine tomorrow 19:55:41 ttx: is there a summit etherpad page yet? 19:55:45 still need to do governance review 19:55:46 yes 19:55:51 jeblair: https://review.openstack.org/#/c/181694/ is an update to zuul-cloner that I think facilitates zuul-cloners use for beaker setup. Other reviews would be great too 19:56:00 https://wiki.openstack.org/wiki/Summit/Liberty/Etherpads 19:56:04 zaro: I don't understand what you mean by that 19:56:19 zaro: are we talking about job logs? or are we talking about logs for the gerrit service itself? 19:56:44 ttx, mordred, clarkb: if you have a moment to create etherpads for the fishbowl sessions and add them to https://wiki.openstack.org/wiki/Summit/Liberty/Etherpads that would be great 19:56:50 jeblair: can do 19:56:51 query on next steps to get refstack.org up and maintained(well, monitored) in infra. krotscheck helped get the webserver together and it's running on refstack.net (davidlenwell's vm) right now. 19:57:40 Rockyg: a puppet module which deploys the service would be the next step 19:57:42 Rockyg: general steps are that we need to get the dns transferred to jbryce if it's not already, then we'll want to get the puppet modules for running the webserver info infra repos 19:57:47 fungi: Already done. 19:57:52 Rockyg: if you could write a spec for infra-specs that would be great :) 19:57:53 woot 19:57:56 mordred: dns looks taken care of now (for refstack.org) 19:58:00 fungi: awesome 19:58:14 kewl. And the foundation has the dns 19:58:20 krotscheck: oh! awesome. i somehow missed the puppet module 19:58:35 fungi: It's not a project yet. Let me go make that pull request. 19:58:42 (it's in github) 19:59:08 Rockyg: so the steps are: infra-spec, create puppet module, then we deploy it 19:59:13 Also something to think about. We need another dns (in one of the reviews, don't know which at the moment) and we might consider coming up with a process for foundation/infra server stuff 19:59:39 clarkb: job logs 19:59:41 And will create the infras spec shortly. Probably not ready before after summit. 20:00:23 Rockyg: well, we have a process for creating servers: http://ci.openstack.org/sysadmin.html#adding-a-new-server 20:00:38 that's kind of our existential purpose :) 20:00:51 * fungi is a machine that makes more machines 20:00:55 I like questioning my existential purpose 20:01:05 jeblair: cool. Just thinking about when the ownership is cross infra and foundation 20:01:07 (it could probably use an update; i'm working on that) 20:01:18 Rockyg: oh, we don't think in those terms 20:01:28 Rockyg: the project owns its own destiny 20:01:35 we enable the project to run itself :) 20:01:38 Oh, good. Perhaps we can discuss more at the summit. 20:01:43 Rockyg: the foundation doesn't have any sysadmins. they're just a foundation. they employ a couple sysadmins and donate them to the infra team 20:01:43 jeblair: ++ 20:01:56 Uh, but we're only stackforge so far? 20:02:11 Rockyg: doesn't matter - we're all one big happy family 20:02:13 ... 20:02:20 and have a ways to go to get to big tent, but we are looking at what we need to get there :-) 20:02:21 o/ 20:02:22 do we have a TC meeting now? 20:02:23 and we should let the projcet run its governance now :) 20:02:23 erm 20:02:28 jeblair: bah 20:02:29 #endmeeting