19:02:10 <fungi> #startmeeting infra
19:02:12 <openstack> Meeting started Tue Aug 16 19:02:10 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:15 <openstack> The meeting name has been set to 'infra'
19:02:23 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:29 <fungi> #topic Announcements
19:02:45 <fungi> #info Reminder: late-cycle joint Infra/QA get together to be held September 19-21 (CW38) in at SAP offices in Walldorf, DE
19:02:47 <fungi> #link https://wiki.openstack.org/wiki/Sprints/QAInfraNewtonSprint
19:02:58 <fungi> note that there's only room for 30 attendees
19:03:01 <rcarrillocruz> i can confirm i'll be attending ^
19:03:06 <rcarrillocruz> got approval for travelling
19:03:11 <fungi> and 22 have already added themselves to that wiki
19:03:29 <fungi> so... if you're going, put yourself on the list as soon as possible or you'll end up on the waiting list
19:03:41 <anteaya> or in the hall
19:03:56 <fungi> i don't know if sap has hallways
19:04:05 <zaro> i will probably drop in for a day
19:04:05 <anteaya> interesting
19:04:11 <fungi> they could be avant-garde
19:04:17 <zaro> does that mean i should put myself on the list?
19:04:22 <anteaya> zaro: yes
19:04:30 <anteaya> fungi: it is possible
19:04:32 <clarkb> ohai
19:04:39 <fungi> zaro: yeah, it sounds like they might be pretty strict about the 30 attendees limit
19:04:57 <nibalizer> o/
19:05:14 <fungi> #topic Actions from last meeting
19:05:14 <zaro> i don't want to take a seat away from someone else that will be there all week
19:05:23 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-08-09-19.04.html
19:05:29 <fungi> there were none
19:05:38 <anteaya> zaro: then you might not get a seat if you try to attend, your call
19:05:51 <fungi> #topic Specs approval
19:06:01 <fungi> we have nothing new this week
19:06:29 <fungi> #topic Priority Efforts: Infra-cloud (rcarrillocruz)
19:06:38 <fungi> says you have an update?
19:06:41 <rcarrillocruz> yeah
19:06:49 <rcarrillocruz> so, it seems we got past all DC and net issues
19:06:57 <rcarrillocruz> i set up the bifrost controller
19:07:06 <rcarrillocruz> and spotted few bugs on bifrost since we last used it
19:07:13 <rcarrillocruz> but with in place fixes
19:07:19 <rcarrillocruz> happy to report i can provision nodes from it
19:07:26 <rcarrillocruz> i showed up a deploy to pabelanger today
19:07:36 <rcarrillocruz> i'm more than happy to show how the thing works to whoever is interested
19:07:54 <rcarrillocruz> and let me know if you want to help out
19:07:57 <rcarrillocruz> that it
19:07:58 <fungi> that's excellent!
19:08:01 <fungi> thanks!
19:08:04 <crinkle> awesome rcarrillocruz
19:08:13 <rcarrillocruz> as a matter of fact
19:08:18 <rcarrillocruz> another redeploy just in time:
19:08:21 <rcarrillocruz> "2016-08-16 19:07:42.901 9824 INFO ironic.drivers.modules.agent_base_vendor [req-03204eef-f9d5-4524-b459-91e307d62a8a - - - - -] Deployment to node d1d4232d-91ec-41d9-9f9b-80350ae9f048 done"
19:08:23 <rcarrillocruz> :-D
19:08:46 <bkero> nice
19:09:09 <clarkb> do we need to revuew changes for puppeting mitaka?
19:09:56 <ianw> rcarrillocruz: cool, could you do some sort of "intro to" session in irc/something at a known time, if a few people are interested?  count me as one...
19:09:58 <rcarrillocruz> i believe crinkle started something  for that?
19:10:14 <rcarrillocruz> ianw: sure, i believe we overlap during my morning for an hour or so
19:10:23 <rcarrillocruz> more than happy to do so in a screen session or something
19:10:45 <jeblair> rcarrillocruz: if there's no private info, you could do this: http://amo-probos.org/post/17
19:11:00 <rcarrillocruz> ah nice
19:11:02 <bkero> rcarrillocruz: https://github.com/yudai/gotty is a great tool for pop-up terminal sharing sessions
19:11:05 <rcarrillocruz> i think it would be fine
19:11:11 <bkero> or that
19:11:21 <rcarrillocruz> no passwords are shown by default on ansible output
19:11:41 <rcarrillocruz> maybe agree a date/time on ML so people can join  ?
19:11:55 <crinkle> if it's not currently on mitaka then i haven't started anything
19:12:41 <crinkle> looks like it is mitaka http://git.openstack.org/cgit/openstack-infra/puppet-infracloud/tree/manifests/controller.pp#n24
19:12:53 <fungi> oh, even better
19:13:08 <crinkle> so nothing to do there for a few more weeks :)
19:13:32 <fungi> anything else we need to cover on infra-cloud?
19:14:03 <rcarrillocruz> nothing from me
19:14:11 <fungi> thanks!
19:14:13 <fungi> #topic Priority Efforts: A Task Tracker for OpenStack (zaro)
19:14:15 <fungi> #link https://review.openstack.org/347486 Enable gerrit/storyboard integration
19:14:29 <fungi> zaro: anything specific about this change?
19:14:34 <zaro> look in commit message of #link https://review.openstack.org/#/c/347486/
19:14:46 <zaro> it hasn't been enabled yet.  we need to enable it
19:15:07 <zaro> should we just enable for all projects?
19:15:45 <fungi> what's the potential impact? other than letting changes for those projects update sb stories and tasks through commit message lines?
19:15:57 <zaro> btw, i think that general task tracker effort is Zara
19:16:05 <Zara> yeah, I'm just watching rn
19:16:06 <fungi> sounds like something we'd just want to enable globally for all repos
19:16:17 <zaro> i think so
19:16:19 <Zara> (since my questions are being covered at this very moment :))
19:16:33 <fungi> zaro: i moved your topic into the priority effort slot because it's related to a priority effort
19:17:06 <zaro> no other impacts i guess
19:17:07 <fungi> Zara didn't mention anything specific on the agenda, so i figured she'd chime in if needed
19:17:29 <fungi> anybody opposed to us enabling storyboard integration for all projects in our gerrit?
19:17:31 <Zara> yeah, I got the impression an infra core needed to do something to enable it? so I'm just watching atm, but if there's something I should do, please let me know
19:18:38 <fungi> i'll take this silence as a tacit agreement
19:18:42 <Zara> \o/
19:18:58 <fungi> #agreed We'll enable Storyboard integration for all projects in our Gerrit
19:19:18 <Zara> so I know what's going on and don't annoy people, who needs to do what to make that happen?
19:19:39 <fungi> #action fungi Add [plugin "its-storyboard"] enabled = true to All-Projects config per https://review.openstack.org/347486
19:19:48 <Zara> oh haha ace
19:19:59 <fungi> i'll also update our sample config in the system-config docs while i'm at it
19:20:04 <Krenair> fungi, I'm back earlier than expected, can talk wiki stuff when it comes up
19:20:08 <Zara> :)
19:20:12 <fungi> Krenair: thanks--we will!
19:20:23 <Zara> fungi: thank you! :)
19:20:24 <fungi> Zara: zaro: anything else on this?
19:20:26 <SotK> \o/
19:20:32 <zaro> nope
19:20:35 <Zara> not from me, yaaaaay I'm so happy! :D
19:20:48 <fungi> #topic Priority Efforts: Zuul v3 (jeblair)
19:20:57 <jeblair> for the nodepool zookeeper work (which i consider on the critical path for zuulv3) we wanted to avoid a feature branch and try to land small stepwise changes.  that's going to be very difficult, and probably not have the intended effect anyway, so i've created a feature/zuulv3 branch of nodepool where we can work on swapping out the builder.
19:21:21 <jeblair> i still think we should adhere to the spirit of that, and once we have the zookeeper builder ready, start running it and merge it into master
19:21:41 <fungi> jeblair: so it's specifically for separate (but related) priority nodepool spec we have?
19:21:59 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html Nodepool: Use Zookeeper for Workers
19:22:07 <jeblair> fungi: yep, i may have misfiled under the agenda :)
19:22:15 <fungi> no worries. close enough
19:22:31 <jeblair> off by one
19:23:04 <clarkb> we are going to tag a release prior as well right?
19:23:09 <clarkb> or did that already happen
19:23:13 <jeblair> clarkb: already done
19:23:24 <fungi> well, we discussed it briefly late last week in #zuul i think, but i'm still in favor now
19:24:16 <jeblair> 0.3.0 is the 'nodepool before zookeeper' tag
19:24:20 <jeblair> we also tagged zuul
19:24:31 <fungi> clarkb: yeah, some weeks ago we tagged a final pre-zk version for nodepool and made an announcement to downstreams that they might ought to pin to that
19:25:09 <fungi> zuul was tagged as 2.5 in keeping with it being the culmination of the "v2.5" ansible-launcher work, even though that meant skipping a couple of minor version numbers to get there
19:25:17 <pabelanger> o/
19:26:14 <jeblair> i also think that we'll be able to switch fairly easily between current nodepool builders and zk when the time comes
19:26:31 <jeblair> (if something goes awry)
19:27:23 <jeblair> [that's all i had on the topic]
19:27:43 <fungi> okay, so we should expect some transitional period where we may go back and forth between the old and new style builders along with zuul v2(.5)?
19:28:05 <jeblair> yeah, that's what i'm thinking.
19:28:20 <fungi> well in advance of zuul v3 running in production anyway
19:28:22 <jeblair> though, hopefully without the back-and-forth.  :)
19:28:34 <fungi> only forth!
19:28:37 * Shrews expects perfect integration the first time
19:28:46 <jeblair> yeah, the nodepool zk builders are still pre-zuulv3
19:29:00 <jeblair> as soon as it's ready we should start using it
19:29:06 <fungi> wfm
19:29:10 <fungi> thanks
19:29:26 <fungi> anybody have other questions on this?
19:29:32 <jeblair> (then make sure we like it before we go all in on zk for the nodepool launcher)
19:29:59 * fungi sallies forth to general meeting topics
19:30:09 <fungi> #topic Gerrit online index testing (zaro)
19:30:21 <fungi> zaro: so what's going on with this?
19:30:35 <zaro> so just to give everyone background on this topic.
19:30:49 <zaro> i am attempting to test the new gerrit online reindex feature on review-dev
19:31:14 <zaro> i initially tried to import nova into review-dev so we can get some prod like data to test with
19:31:30 <zaro> but that really just wasn't working due to how the importer plugin works
19:32:00 <zaro> fungi suggested we just pump changes into review-dev to load up it up for testing, so just fake data
19:32:07 <fungi> lack of corresponding accounts for change owners/reviewers and whatnot, if memory serves?
19:32:15 <zaro> i've done that, so now review-dev has about ~100k changes
19:32:25 <anteaya> zaro: awesome
19:32:27 <zaro> fungi: yes, and plugin had bugs
19:32:34 <fungi> bugs? impossible ;)
19:32:49 <zaro> and i've tested with just me poking at review-dev while reindexer is runnning
19:33:02 <zaro> everything seems good but that's not the test i was after.
19:33:27 <zaro> i wanted to run online reindex and have multiple users use review-dev while it's running.
19:33:49 <anteaya> how long does the reindex take with 100k changes?
19:33:52 <zaro> but problem now is review-dev is low on resources due to much more changes that got pumpped in
19:34:02 <zaro> anteaya: about 1.5-2hrs
19:34:09 <anteaya> thanks
19:34:28 <zaro> if we want that better test then we need to up the resources on review-dev
19:34:44 <fungi> so next suggestion i think was that we could rebuild review-dev on a bigger instance
19:34:58 <fungi> keep the db/homedir from the old server
19:35:13 <zaro> more RAM and more disk please!
19:35:19 <zaro> ohh cpu too
19:35:19 <anteaya> how much more?
19:35:46 <fungi> yeah, that's the main question
19:35:55 <zaro> if i had a list of flavors to pick from that would be great
19:36:12 <fungi> they mostly double except after 15 it goes to 30
19:36:17 <fungi> for gigabytes of ram
19:36:37 <zaro> maybe just half of whatever's on prod would be enough
19:36:48 <zaro> i think currently it's only got 4gig
19:37:00 <fungi> looks like current review-dev is a 4gb flavor
19:37:02 <fungi> yeah
19:37:26 <anteaya> prod is 30 right?
19:37:27 <zaro> is half of prod asking too much
19:37:28 <zaro> ?
19:37:33 <fungi> review.o.o (prod) is 60gb
19:37:35 <zaro> prod is 60gg
19:37:39 <anteaya> ah
19:37:56 <zaro> and disk space on review-dev is really low
19:38:16 <anteaya> I personally have no objection
19:38:22 <anteaya> it is based on quota, yeah?
19:38:27 <anteaya> have we quota available?
19:38:34 <fungi> though keep in mind we only really bumped production because we thought it would help the java gc memory pressure. instead that seems to just be a leak so no amount of extra ram is going to help
19:39:00 <zaro> yeah, well aware of that.
19:39:03 <fungi> disk space doesn't really change on the new flavors, we ought to just move ~gerrit2/review_site into a cinder volume
19:39:15 <zaro> ++ there
19:39:19 <fungi> we can make that as big as we want, effectively
19:39:40 <fungi> (or at least bigger than we'll need for this purpose)
19:40:34 <fungi> okay, so while 30gb is a bit much, perhaps we could rebuild again on a smaller flavor once you're done trying to heavily load-test it?
19:40:47 <zaro> sure.
19:41:12 <fungi> also, any infra-core volunteers to launch the new review-dev server and add a cinder volume to it?
19:41:56 <rcarrillocruz> Me
19:42:00 <zaro> also i have tested the project change scenario with online reindex and it works like a charm.
19:42:15 <zaro> project change/ project name change
19:42:28 <rcarrillocruz> fungi: ^
19:42:32 <anteaya> zaro: that is good to know, thank you
19:42:54 <ianw> fungi: i'm new to it, but can help with maybe some pointers
19:43:05 <fungi> #action rcarrillocruz launch new 30gb review-dev and add an appropriately sized cinder volume for ~gerrit2/review_site
19:43:33 <fungi> ianw: maybe you and rcarrillocruz have some overlap time at the end of your day and early on his
19:43:44 <rcarrillocruz> Yup
19:43:48 <fungi> definitely team up. we have stuff pretty heavily documented
19:44:12 <fungi> anything else for the gerrit online reindex testing?
19:44:20 <ianw> cool
19:44:21 <zaro> nope.
19:44:39 <fungi> #topic wiki status update (jpmaxman, Krenair, fungi)
19:44:59 * Krenair waves
19:45:16 <fungi> thanks to jpmaxman's great testing and instructions, i manually upgraded production wiki.openstack.org in-place to mw 1.27 and ubuntu trusty on friday
19:45:32 <anteaya> thank you
19:45:37 <Zara> \o/
19:45:41 <jpmaxman> :) how's the spam?
19:45:47 <fungi> seems to be working well so far. hasn't put a stop to spam but the newer captcha for confirmedit seems to have drastically reduced it
19:45:48 <jpmaxman> haven't checked if it actually worked
19:46:11 <fungi> we got several new accounts adding spam pages on saturday, and another one a few hours ago
19:46:22 <fungi> but previous weeks that would have been an order of magnitude higher
19:47:00 <fungi> since the upgrade there have been more days with no new spammers than with them
19:47:15 <jpmaxman> cool - so some spam is still getting through?
19:47:24 <fungi> jpmaxman: yeah, just not nearly as much
19:47:27 <Krenair> does this spam (that still gets through) contain links?
19:47:37 <jpmaxman> ok well it is a start!
19:47:51 <fungi> same kind of spam though (phone numbers for scammers claiming to provide software/hardware support)
19:47:59 <fungi> Krenair: nah, not for the most part
19:48:04 <fungi> it's phone number spam
19:48:07 <Krenair> ah
19:48:15 <jpmaxman> I mean if it is really a person - I guess there isn't a lot to be done
19:48:15 <Krenair> yeah we have that on mediawiki.org sometimes
19:48:44 <fungi> also curious to try turning file uploads back on and seeing if confirmedit is covering those with the captcha now
19:48:46 <anteaya> stop bots from indexing the wiki
19:49:03 <fungi> anteaya: that's an option we're still reserving as a possibility
19:49:04 <anteaya> that is still an option
19:49:08 <anteaya> right
19:49:21 <Krenair> heh
19:49:23 <jpmaxman> yeah I mean ideally you want people to find the content through a google search
19:49:29 <anteaya> and past experience shows that vitually eliminates the spam
19:49:29 <fungi> though i'd also like to see if we can give the stopforumspam blacklist extension a shot
19:49:40 <anteaya> that is fair
19:50:33 <fungi> basically they track and maintain a live list of ip addresses where they see abusive comments originating from, and the extension refuses to allow page edits from those ip addresses
19:50:36 <anteaya> jpmaxman: what we would really like is for people to find docs.openstack.org and git.openstack.org via search engines
19:50:37 <jpmaxman> ahh that reminds me
19:50:39 <Krenair> imagine if wikipedia.org did that anteaya
19:50:47 <jpmaxman> There is an additional configuration option for this module, $wgReCaptchaSendRemoteIP (default: false), which, if set to true, sends the IP address of the current user to a server from Google while verifying the CAPTCHA
19:50:58 <anteaya> Krenair: the business model is different
19:51:13 <fungi> jpmaxman: oh, so maybe nocaptcha has some built-in ip address blacklisting too?
19:51:20 <jpmaxman> fungi we can try setting that to true - seems similar to stopforumspam blacklist but baked into the nocaptcha
19:51:25 <jpmaxman> yeah we just didn't enable it
19:52:09 <fungi> worth a try. the risk obviously is we're disclosing ip addresses to a third party, but... we'd be doing that via things like google analytics too
19:52:13 <anteaya> Krenair: one of our options is not having a wiki
19:53:02 <fungi> anteaya: Krenair: right. on the spectrum of options we're evaluating, having a wiki that isn't indexed by major search engines might still be a good compromise between having a wiki full of spam or having no wiki at all
19:53:06 <jpmaxman> so next steps - Krenair I still need to go through your commits one thing I brought up with fungi was apache config changes for hosting the site in /srv/mediawiki - I wasn't sure if there was a reason for that - would be able to keep more defautls if it was in /var/www
19:53:15 <anteaya> fungi: right
19:53:30 <fungi> jpmaxman: oh, right, i was still composing an e-mail reply
19:53:38 <Krenair> jpmaxman, isn't that the existing place puppet puts it?
19:53:49 <Krenair> I don't think I changed that
19:53:58 <fungi> we mostly try to follow the filesystem hierarchy standard and not write non-packaged, static content into /var
19:54:04 <jpmaxman> I'm not sure - just on the existing server it is /srv/mediawiki
19:54:26 <fungi> pretty much all our apache vhosts are served out of /srv/something on our servers
19:54:45 <jpmaxman> ok - that's fine - just need to consider that for Krenair puppet config
19:54:54 <fungi> which we often (as will also be the case with wiki.o.o) mount from a separate cloud block storage device
19:55:13 <jpmaxman> ubuntu apache package assumes /var/www and adds some config options to that directory which need to then be applied to the alternate location
19:55:13 <Krenair> You're referring to this change jpmaxman? https://review.openstack.org/#/c/352033/1/templates/apache/mediawiki.erb
19:55:53 <jpmaxman> sorry Krenair - I'm just talking generally - I apologize I haven't taken a dive into the changes
19:56:11 <jpmaxman> if it's already considered then just ignore me :)
19:56:17 <Krenair> It's not considered
19:56:19 <Krenair> It's not relevant
19:56:35 <Krenair> The existing directory puppet uses is under /srv
19:56:54 <Krenair> Changing MW and Ubuntu versions doesn't affect that
19:57:15 <fungi> i think the underlying point was that when updating our apache vhost config in production to cope with the apache 2.2 to 2.4 move there were (typical) updates needed to directory allow/grant and options
19:57:21 <jpmaxman> ok I had to make some changes going 2.2 --> 2.4 - you have one in here -                         Require all granted
19:57:31 <jpmaxman> I'll double check if anything else is necessary
19:57:34 <jpmaxman> it may not be
19:58:18 <fungi> jpmaxman was suggesting that using /var/www instead of /srv/mediawiki might make that unnecessary, but i'm pretty sure 1. it won't since we still need some pretty specific directory blocks for various mediawiki bits, and 2. we would rather stick with /srv for other reasons i already mentioned
19:58:40 <fungi> jpmaxman: Krenair i have a diff i can link, just a sec, before the meeting ends
19:59:12 <jpmaxman> fungi: agreed.  The only reason it is not necessary is ubuntu package for apache puts it in the "upstream" conf for /var/www
19:59:28 <fungi> #link http://paste.openstack.org/show/558525 apache vhost diff for 2.2 to 2.4 update
19:59:30 <jpmaxman> it's more important I'd think to keep things consistent with your other servers
19:59:56 <anteaya> thanks for the meeting fungi
19:59:58 <fungi> anyway, we're out of time
20:00:08 <fungi> oh, also before i endmeeting, reminder
20:00:08 <zaro> I have the verify-status plugin (publishing test metadata to gerrit) on #link http://138.68.20.113:8080/#/c/2/ if anyone is interested to see how it works.
20:00:19 * amrith waits for the doors to open
20:00:23 <Krenair> fungi, um.
20:00:24 <fungi> #link https://review.openstack.org/#/q/topic:wiki-upgrade+is:open changes for wiki puppetization
20:00:27 <fungi> thanks for that Krenair!
20:00:34 <fungi> #endmeeting