19:01:00 <fungi> #startmeeting infra
19:01:02 <openstack> Meeting started Tue Nov 10 19:01:00 2015 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:06 <openstack> The meeting name has been set to 'infra'
19:01:09 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:14 <ruagair> *\o/*
19:01:14 <fungi> #topic Announcements
19:01:15 <asselin_> 0/
19:01:20 <fungi> nibalizer would like to identify himself as a potential mentor to anyone looking to get started with infra
19:01:23 <nibalizer> o/
19:01:35 <AJaeger> o/
19:01:41 <fungi> nibalizer: i just introduced your announcement before you /join'ed
19:01:47 <jesusaurus> o/
19:01:49 <AJaeger> great, nibalizer !
19:01:51 <fungi> please elaborate
19:01:54 <nibalizer> awesome
19:01:57 <Clint> o/
19:02:08 <nibalizer> basically just if anyone is lurking and identifies as 'new' and would benefit from some mentoring, im available
19:02:13 <crinkle> o/
19:02:17 <fungi> i think it's admirable, and i hope we all aspire to mentor people getting started with infra work
19:02:22 <nibalizer> i think we all want to help everyone, but there is a lot of noise
19:02:33 <nibalizer> and work to do, etc
19:02:39 <fungi> yep!
19:02:40 <nibalizer> thats it
19:02:48 <jeblair> nibalizer: ++
19:02:49 * ruagair is enroute to the airport and boarding during this meeting. So may be sluggish to respond.
19:02:52 <fungi> so nibalizer is our go-to mentor! ;)
19:02:58 <maiteb> uh, nice... I would like some mentorship actually :)
19:02:58 <pabelanger> Agreed. Infra has a high learning curve, anything we can do to help new people works for me
19:02:59 <dramalho> fungi: Sounds good... +2 :)
19:03:06 <Zara> I'd like to help down the line but I'm not sure I know enough yet.
19:03:29 <jhesketh> nibalizer: cool stuff :-)
19:03:31 <nibalizer> maiteb: cool, after the meeting lets chat a bit
19:03:45 <maiteb> great nibalizer :)
19:03:53 <fungi> #info Spencer Krum (nibalizer) has volunteered to mentor new contributors to the OpenStack Project Infrastructure team
19:04:13 <dramalho> nibalizer: I would like some mentorship actually, too :)
19:04:33 <fungi> #topic Actions from last meeting
19:04:34 <bkero> nibalizer is popular
19:04:38 <fungi> there were none, all executed successfully!
19:04:43 <dramalho> haha
19:04:48 <jeblair> yay us!
19:04:56 * Clint golfclaps.
19:05:03 <fungi> post-summit is always such an amusing time
19:05:09 <fungi> #topic Specs approval
19:05:16 <fungi> the cycle priority updates are now reflected on the specs page
19:05:17 <fungi> i think
19:05:23 <fungi> i had to retrigger the job
19:05:26 <nibalizer> excellent
19:05:28 <fungi> anyway, if they're not, they soon will be
19:05:34 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/#priority-efforts
19:05:40 <fungi> #info Approved change to update priorities for the Mitaka cycle
19:05:47 <fungi> phschwartz has a spec proposed to extend openstackci for downstream ci systems b
19:05:49 <fungi> eyond just third-party ci strung from our (openstack's) gerrit instance
19:06:04 <fungi> hrm, you can have the stray linefeed as a bonus
19:06:09 <fungi> no extra charge
19:06:16 * nibalizer pockets
19:06:26 <fungi> #link https://review.openstack.org/239810
19:06:28 <phschwartz> fungi: I will take what I can get
19:07:08 <fungi> just to confirm, this is ready for final council voting?
19:07:19 <phschwartz> from my point of view yes.
19:07:26 <nibalizer> i think so
19:07:39 <clarkb> I thought we already had a spec for this...
19:07:48 <clarkb> "make puppet modules reconsumable"
19:07:56 <clarkb> anyways carry on
19:08:08 <jeblair> clarkb: apparently this part was just in our heads :)
19:08:38 <fungi> can't hurt to document the plan more thoroughly, as long as it doesn't contradict what we may have already said we would do
19:08:45 <fungi> #info Council voting will remain open for the "Add extension to openstackci for next phase of work" spec until 19:00 UTC Thursday, November 12
19:09:18 <fungi> #topic Priority Efforts: Gerrit 2.11 Upgrade
19:09:41 <fungi> in case it seems like i was rushing through the first part of the meeting, it's because i expect us to spend a while today talking about this
19:09:47 <fungi> #link https://etherpad.openstack.org/p/mitaka-infra-gerritdevelopment
19:09:52 <fungi> #link https://etherpad.openstack.org/p/test-gerrit-2.11
19:10:03 <fungi> zaro: how are things shaping up for the upgrade next week?
19:10:41 <fungi> (or anybody who's been working with zaro on the preparations i guess)
19:10:44 <jeblair> if zaro is unresponsive it may be because he is at gerrit hackathon today....
19:10:51 <fungi> aha!
19:10:56 <fungi> very good point
19:11:05 <clarkb> zaro found the double // root of url openid redirect bug
19:11:09 <asselin_> I have a test 3rd party ci that's commenting 'noop' patches to review-dev
19:11:13 <clarkb> fix has been proposed upstream and is in place on review-dev
19:11:37 <fungi> clarkb: oh, excellent. do you happen to have a link? i'm curious where it turned out being
19:11:37 <clarkb> someone said that zuul needed updates to handle new events?
19:12:05 <jeblair> i thought we merged those changes to zuul ages ago?
19:12:07 <fungi> i assume something in the openid support bits was appending a / to the return url being sent with the authentication redirect
19:12:26 <clarkb> #link https://gerrit-review.googlesource.com/#/c/72105/
19:12:33 <clarkb> looks like that particular change was abandoned to move it to stable branch
19:12:51 <clarkb> I am guessing they will merge that back up into master?
19:12:58 <fungi> i thought we at least merged new-in-2.9/2.10 events support in zuul and jeepyb prior to the upgrade attempt in may
19:12:59 <jeblair> on topic:gerrit-upgrade i see a fix for toggle-ci and a fix to jeepyb still pending.
19:13:08 <jeblair> fungi: agreed
19:13:21 <jeblair> fungi: the jeepyb change references 2.11
19:13:22 <clarkb> maybe they weren't installing from HEAD like we do
19:14:39 <clarkb> change screen 2 feedback has been mostly positive
19:14:57 <clarkb> mordred: have you written the db rollback yet?
19:14:58 <fungi> anyway, hopefully all outstanding changes needed for the upgrade are using the gerrit-upgrade topic
19:15:05 <fungi> #link https://review.openstack.org/#/q/status:open+topic:gerrit-upgrade,n,z
19:15:15 <jeblair> i wonder if we need 209906 for 2.11...
19:15:23 <jeblair> the sortkey pagination deprecation
19:15:40 <fungi> let's all make sure staying on top of those is a particularly high priority so that we're ready in time for the maintenance a week from tomorrow
19:16:02 <clarkb> I will carve out time after lunch to get through the current list of changes
19:16:11 <jeblair> #action jeblair investigate whether 209906 is needed for gerrit 2.11
19:16:16 <clarkb> someone should probably send mail to the list about the upgrade?
19:16:25 <nibalizer> I did that at the summit
19:16:27 <nibalizer> or very nearly after
19:16:29 <clarkb> oh perfect
19:16:30 <jeblair> nibalizer sent an initial... maybe time for a followup?
19:16:35 <clarkb> ya followup would be good
19:16:40 <jeblair> maybe a 1 week notice tomorrow?
19:16:52 <clarkb> ++
19:16:58 <AJaeger> a reminder in time so that it comes into the weekly newsletter?
19:17:04 <fungi> #action nibalizer send one-week reminder for scheduled maintenance on the 18th
19:17:13 <clarkb> if anyone can get in front of mordred finding out about the DB rollback prep would be good
19:17:18 <clarkb> since I know he wanted that done before the upgrade
19:17:29 <nibalizer> ok
19:17:36 <fungi> nibalizer: i can #undo if you don't want that one
19:17:51 <nibalizer> fungi: im good with it
19:17:52 <fungi> though continuity is fairly useful on those announcements
19:17:55 <fungi> excellent
19:18:10 <clarkb> this reminds me to send mail to the list saying elasticsearch upgrade is (mostly) done
19:18:22 <nibalizer> fyi my plans changed and I will be travelling for the downtime, but I still have the day blocked off to perform the upgrade
19:18:41 <nibalizer> so it will be nibz-from-hotel instead of nibz-from-home (which should be transparent to this team)
19:18:56 <fungi> as previously indicated, i'll be gone all next week (starting friday actually) so will miss all the fun
19:19:13 <fungi> but i'm looking forward to returning to a brand new gerrit 2.11 world
19:20:31 <clarkb> oh
19:20:39 <fungi> seems like we exhausted this topic much quicker than i anticipated
19:20:42 <clarkb> has anyone run git revie wagainst review-dev?
19:21:04 <fungi> i did for 2.10 a lot but haven't tried with 2.11 yet
19:21:07 <clarkb> thats probably overly paranoid
19:21:16 <clarkb> I can go ahead and do that today when I am done reviewing changes
19:21:26 <nibalizer> jeblair: seems to have posted new changes on nov 7th, so probably
19:21:39 <fungi> clarkb: yeah, that's not a terrible idea
19:21:55 <jeblair> nibalizer: ?
19:22:01 <fungi> #action clarkb double-check git-review interactions with gerrit 2.11 on review-dev.openstack.org
19:22:11 <clarkb> jeblair: have you run git review against latest review-dev deployment?
19:22:13 <nibalizer> wait nevermind, I misread
19:22:16 <jeblair> nibalizer: you might be looking at updated rather than created
19:22:18 <jeblair> clarkb: nope
19:22:20 <nibalizer> ya exactly
19:22:22 <clarkb> kk I will do it
19:24:08 <fungi> okay, anything else on this topic?
19:24:21 <nibalizer> not from me
19:24:39 <fungi> #topic Priority Efforts: maniphest migration
19:24:46 <fungi> ruagair has a bunch of updates in the meeting agenda, but i'll let him iterate through them if he's here. otherwise i can just cut and paste them into the meeting myself
19:24:58 <ruagair> I'm here
19:25:11 <ruagair> nearly at airport though.
19:25:14 <fungi> awesome. anything there you need to highlight?
19:25:39 <clarkb> I would be curious to know if any progress was made on the auth front
19:25:42 <ruagair> I have done low priority changes that need more approve ers
19:26:01 <ruagair> A small amount Clarkb
19:26:03 <fungi> #link https://review.openstack.org/#/q/status:open+topic:maniphest,n,z
19:26:16 <ruagair> Not as much as I expected.
19:26:30 <clarkb> ruagair: ok, I played with mod_auth_openid myself a bit and ran into what I think are ssl and dns issues
19:26:44 <clarkb> it appears that you need to have a fairly correct deployment for the id server to talk to you
19:27:11 <ruagair> I am expecting that today I'll have phab taking to openstackid via cauth.
19:27:18 <clarkb> neat
19:27:31 <ruagair> I thought that yesterday though.
19:27:40 <clarkb> :) I know the feeling
19:28:15 <fungi> fwiw, the puppet-openstackid module should be sanely deployable on a test instance if you need to inspect the other end of the interaction
19:28:16 <ruagair> I've not yet started with mod-auth-openid
19:28:48 <ruagair> thanks fungi, I'll make use of that.
19:29:03 <clarkb> ruagair: if I get it working independently I can pass that info along
19:29:20 <ruagair> please do.
19:30:43 <ruagair> I'm in touch with a cauth contributor too, who's been helpful.
19:31:09 <fungi> anything else to relay on this front?
19:31:26 <ruagair> No
19:31:37 <ruagair> hit me with a questions :-)
19:31:58 <Zara> ruagair: have you had an updated spec for what people want from a task tracker?
19:32:11 <ruagair> No.
19:32:29 <ruagair> Had a long talk with ttx but no updated spec.
19:32:49 <Zara> heh, okay, please update me if/when you get one! :)
19:32:53 <fungi> that would be good to get nailed down so we have something to measure the deployment against as well
19:32:59 <ruagair> Will do.
19:33:00 <Zara> thanks
19:33:07 <ruagair> exactly fungi
19:33:40 * ruagair is now purporting
19:33:52 <ruagair> airporting
19:33:52 <fungi> okay, this was a remarkably short meeting. i'll switch to open discussion unless anyone else has last-minute topics they want to wedge in
19:34:27 <greghaynes> short meetings are best meetings
19:34:37 <phschwartz> greghaynes: ++
19:35:02 <fungi> #topic Open discussion
19:35:34 <jeblair> i sent an email summary about the gerrit user summit to the list
19:35:45 <Clint> https://review.openstack.org/#/c/219372/ still lacks review
19:35:48 <nibalizer> I took a stab at setting up the polygerrit app mentioned in jeblair's email http://polygerrit.nibalizer.com/q/status:open
19:35:55 <clarkb> nibalizer: and I upgraded elasticsearch to version 1.7.3 yesterday. Killing some long standing tech debt
19:36:02 <jeblair> #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/078945.html
19:36:12 <jeblair> #link http://polygerrit.nibalizer.com/q/status:open
19:36:17 <jeblair> nibalizer: which is way cool :)
19:36:22 <fungi> taron: if you're looking for a template to use for announcing availability of http://codesearch.openstack.org/ you can consider cribbing from pabelanger's grafana.o.o announcement
19:36:29 <fungi> #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/079082.html
19:36:36 <nibalizer> ya, that needs a big 'I didn't write this, this is beta, teh source is <here>' sign on it
19:36:59 <clarkb> need to fix CORS for admin dashboards, fix e-r which is hitting pyelasticsearch bug, then next steps are updating logstash to current version, updating kibana, and doing rolling upgrades from precise to trusty
19:36:59 <nibalizer> the puppet apply stuff is going really well
19:37:09 <clarkb> clear out a lot of the ELK cruft that has accumulated
19:37:14 <fungi> nibalizer: large text saying "file bugs <here>, submit fixes <there>"
19:37:20 <clarkb> then maybe next cycle we update to ES 2.0
19:37:28 <jeblair> i believe i'm very close to being able to put a very limited number of nodes from bluebox and ovh into production
19:37:33 <phschwartz> nibalizer: let me know if there is any other help you need for the puppet-apply stuff
19:37:34 <clarkb> jeblair: !
19:37:39 <greghaynes> jeblair: nice!
19:37:40 <pabelanger> fungi: I really should have floated that by infra-roots before posting, sorry about that
19:37:54 <jeblair> like, not at 'big splash announcement heavy use' yet, but more 'dip toes into shallow end'.  :)
19:38:26 <jeblair> also, in both cases, i think our cpu needs are going to end up causing us to have fewer nodes than we expected
19:38:56 <jeblair> but we'll see, things are still in flux
19:39:07 <fungi> pabelanger: no worries. the announcement was fine and timely
19:39:22 <fungi> thanks for working through that deployment!
19:39:33 <jeblair> pabelanger: yay! :)
19:39:39 <pabelanger> np, was fun and learning experience
19:40:13 <pabelanger> On that note... http://stackalytics.openstack.org is live too
19:40:31 <pabelanger> for the moment, just doing some tuning of the server
19:40:39 <fungi> jeblair: so the lesson here is that the cpu performance provided by rackspace and hpcloud are not actually representative of the broader ecosystem of public clouds?
19:40:52 <pabelanger> but I think this are actually good after talking to ilyashakhat
19:41:06 <jeblair> pabelanger: yeah, might be good to fully understand the pegged uwsgi process before announcing?
19:41:18 <fungi> pabelanger: no longer feeling the urge to rip out apache? ;)
19:41:23 <phschwartz> fungi: I think it is more along the lines that rackspace and hpcloud have been around longer so they have larger infrastructure to provide more resources from
19:41:45 <jeblair> fungi: i think rackspace is exceptional, and hpcloud is par; i think the thing we're learning is actually more about private/public....
19:41:46 <pabelanger> fungi: Ya, might have been hasty on that
19:42:09 <fungi> phschwartz: well, the performance per vcpu i mean
19:42:15 <jeblair> i'm going to write up some changes to the 'cloud requirements' thing to elaborate on that soon
19:42:31 <dramalho> We are currently working on adding some tests for puppet modules and fixing them as we find bugs. Is this important ?
19:42:55 <fungi> writing tests and fixing bugs is always important
19:43:18 <jeblair> fungi, phschwartz: but really short version: if we are sole occupants, we are bad neighbors for ourselves, so we need less oversubscription; if we are distributed in a public cloud, we can handle more oversub.
19:43:18 <clarkb> I can say the thing that turned me off from reviewing many of those changes is they didn't seem to test how we/others use the modules
19:43:29 <fungi> as long as the bugs aren't simply bugs because the tests say they are, but rather the tests are exercising the system in such a way as to expose legitimate issues
19:43:34 <clarkb> dramalho: for me at least I would like to see the tests focus on how people are using the modules to start
19:43:45 <clarkb> fungi: right
19:44:03 <fungi> jeblair: that makes a lot of sense
19:44:14 <pabelanger> jeblair: I think it was because we didn't fully index the data properly. And because we didn't have a fully data dump, it was reordering things every hour.  However, going to watch it and try to a better reason
19:44:21 <yolanda> clarkb, actually these tests provided some value, discovering problems with races, or incompatibilities with different distros
19:44:42 <clarkb> yolanda: right but the focus should be on testing how its used first
19:45:04 <clarkb> bceause while getting support for $distro is nice if we aren't deployed on that distro the tests failing there is less important than actual functionality as deployed being tested
19:45:07 <fungi> agreed, exposing portability issues doesn't necessarily mean you're finding bugs
19:45:42 <yolanda> what do you suggest? integration testing?
19:45:48 <zaro> o/
19:46:04 <yolanda> as it's a test with modules, the near we can go is providing default values that resemble more to system-config
19:46:06 <fungi> porting a module to a different platform than it was originally written for is generally an effort unto itself
19:46:47 <yolanda> fungi, yes, actually that concern was raised by some members of the community , but that's a different topic
19:46:48 <fungi> more of a feature than a bug fix
19:47:08 <clarkb> yolanda: I would suggest testing functionality we use
19:47:09 <yolanda> well, it's true that they discovered races, that you only can see when deploying a module for first time
19:47:24 <clarkb> for example in httpd module there were proxy tests written before "apply this vhost" tests
19:47:30 <clarkb> same for redirects iirc
19:47:41 <clarkb> the vast majority of our use of that module is "here is a vhost apply it"
19:47:42 <clarkb> that should be test 1
19:47:56 <nibalizer> fungi: some of whats happening is the modules have been written for precise, but the tests run on trusty, so little things are different
19:48:05 <nibalizer> i think thats what they are refering to when they say portability issues
19:48:08 <yolanda> dramalho and tw guys, is something you want to take care?
19:48:29 <yolanda> nibalizer yes, that was number one portability issue :)
19:48:30 <fungi> nibalizer: yep, i would agree those are portability issues (precise and trusty are different platforms)
19:48:51 <nibalizer> its not like we're trying to get the modules to work on solaris :)
19:48:54 <fungi> however we want to port our use of those from precise to trusty, so testing them with trusty and fixing bugs discovered there is helpful
19:49:04 <nibalizer> exactly
19:49:37 <fungi> i thought most of that had been solved already, but i guess only for more popular modules
19:49:37 <yolanda> there was an special interest from RedHat community to make this module work on CentOS/RHEL, but that's a different topic that we should talk in another meeting. I proposed them to raise a topic on the agenda
19:50:22 <fungi> if we're going to plan to "support" (whatever that means) a module on a given platform, we should test it on that platform
19:51:15 <yolanda> yes, i agree
19:51:46 <yolanda> so i wouldn't discourage TW team to do functional testing, but improve the ones they are doing
19:52:06 <clarkb> yolanda: my feedback would be to start with the most valuable tests first
19:52:32 <fungi> even though it's tempting to start with the easiest tests to write first
19:52:40 <clarkb> (you could probably just copy pasta the relevant manfiest bits from the various manifests and put that in the beaker rspec fixtures)
19:53:30 <dramalho> yolanda: we need to see what tests we already did and the list of modules that we are thinking to write tests and see which one will be more valuable
19:54:14 <yolanda> so dramalho, glauco, have done a decent job there, testing the basic features of the modules is important. I agree that extending use cases, to properly test some features, is something we will need
19:56:16 <fungi> thanks for working on that!
19:56:20 <maiteb> yolanda: for example, dramalho and I are now testing puppet-etherpad_lite, what do you guys think that we need to focus on? only basic testing?
19:57:14 <yolanda> from my point of view is important that all functional tests ensure the basic features of the module are working
19:57:15 <clarkb> maiteb: I would start from the openstack_project::etherpad manifest
19:57:22 <yolanda> services are up, files are created, cronjobs are on place
19:57:40 <crinkle> i think both things are valuable - minimal test using mostly default parameters (how someone new might be using the module) and using openstack_project in system-config (how infra actually uses it)
19:58:04 <fungi> i agree with clarkb. we have example production use cases already written up in our system-config repo
19:58:33 <fungi> in fact, we have two for that module (one for etherpad-dev as well)
19:58:43 <dramalho> fungi: Cool... sounds good...
19:59:15 <maiteb> crinkle: yes, that makes sense...
19:59:17 <fungi> so basically, yeah, what crinkle said too
20:00:25 <fungi> we're out of time--thanks everyone!
20:00:29 <fungi> #endmeeting