19:02:05 <fungi> #startmeeting infra
19:02:06 <openstack> Meeting started Tue Jan  3 19:02:05 2017 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:09 <openstack> The meeting name has been set to 'infra'
19:02:14 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:21 <fungi> #topic Announcements
19:02:33 <fungi> looks like i don't have any for this week
19:02:36 <fungi> as always, feel free to hit me up with announcements you want included in future meetings
19:02:45 <fungi> #topic Actions from last meeting
19:02:49 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-12-20-19.03.html
19:02:55 <fungi> "1. (none)"
19:02:59 <fungi> good work!
19:03:03 <zaro> O/
19:03:05 <fungi> i have to admit, i did a lot of that
19:03:11 <fungi> not ashamed
19:03:28 <fungi> #topic Specs approval
19:03:35 <fungi> we don't seem to have anything new up this week
19:03:55 <fungi> #topic Priority Efforts: Docs Publishing via AFS (AJaeger)
19:03:58 <fungi> #link http://lists.openstack.org/pipermail/openstack-docs/2016-December/009430.html production docs.o.o cut-over plan
19:04:08 <fungi> we were talking about this in irc yesterday, and it feels like we're ready to forge ahead here
19:04:29 <fungi> the plan looks pretty solid
19:05:01 <jeblair> i was just catching up on that this morning
19:05:08 <AJaeger> yes, I think we're ready...
19:05:35 <jeblair> i had one question -- why make docs-archive.o.o?  is files.o.o not sufficient for being able to retrieve a file in the (unlikely) event that we need it?
19:05:36 <AJaeger> updated plan at http://lists.openstack.org/pipermail/openstack-docs/2017-January/009436.html
19:06:00 <AJaeger> jeblair: you cannot really follow links
19:06:27 <AJaeger> all links to / will go to files.o.o instead of to the docs root
19:06:32 <jeblair> okay, but i thought the archive was so that we can pull a file out and copy it over if we find it's missing
19:06:32 <fungi> AJaeger: are they absolute, not relative?
19:07:08 <AJaeger> it really depends - and some have docs.o.o hardcoded
19:07:13 <jeblair> creating a new vhost to serve obsolete content seems, at best, unecessary, and at worst, actually counter productive
19:08:06 <fungi> i guess the primary need is to have a copy of the files in case we forgot anything after we've told rackspace to tear down the cloudsites account. i'm less concerned about giving it a dedicated vhost and custom 404 forwarding but those also sound pretty cheap to add
19:09:13 <fungi> it does also seem like we should stop serving a dedicated site for the stale archive after some period of time, even if we do continue to keep copies of the files around
19:09:52 <fungi> mainly to avoid confusion like jeblair mentions
19:09:54 <jeblair> okay.  it's a significant change to the plan.  i'd love it if we had a good reason for it, and a plan for retiring it, etc.  but i'm not going to veto it if someone wants to do that (extra) work.
19:11:05 <fungi> i'm mostly ambivalent, though curious to see arguments for why the files.o.o interface is really insufficient
19:11:07 <AJaeger> we can move forward without the extra vhost - and discuss the need of a vhost with the rest of the docs team.
19:11:14 <jlvillal> o/
19:11:21 <jeblair> (like, to me, the fact that links don't work is not a problem as i never envisioned the purpose of this to be that we would say "oh, you're looking for that doc?  it's hosted on docs-archive.openstack.org"
19:11:48 <jeblair> i thought the purpose was to say "oh, crap, we're missing a doc, let me find it in the archive and manually copy it over")
19:12:31 <AJaeger> my intention is indeed manually copying it over
19:12:58 * AJaeger checks some old emails
19:13:05 <jeblair> if you make docs-archive.o.o, make sure you make a fully-exclude robots.txt
19:13:38 <fungi> right, we really really really don't want that appearing in search engines
19:13:54 <AJaeger> indeed
19:15:04 <fungi> okay, so are we at least agreed we can move forward with steps 1 and 2 for now, and discuss the necessity of step 3 with the docs team?
19:15:36 <AJaeger> yes
19:15:42 <jeblair> yes for 1...
19:16:04 <fungi> what other blockers do we have for 2?
19:16:04 <jeblair> for 2 -- i pushed up a patch to do a vos release in cron so that we could serve docs from a readonly volume
19:16:12 <fungi> oh, that hasn't merged yet?
19:16:16 <jeblair> i'm not sure if it landed
19:16:24 <fungi> pretty sure i reviewed it, but checking
19:16:27 <jeblair> AJaeger: i think you were fixing it up?
19:16:46 <jeblair> #link https://review.openstack.org/400887
19:16:46 * AJaeger checks
19:16:51 <fungi> i seem to have new-year amnesia so i don't trust my pre-holiday memories at this point
19:17:00 <jeblair> yeah, working through some fog myself :)
19:17:10 <AJaeger> jeblair: yes, I fixed it
19:17:19 <jeblair> we may want to land that and create the replica volumes before doing #2
19:17:23 <fungi> okay, looks like i did review it
19:17:49 * AJaeger updates plan and adds that change
19:18:17 <jeblair> (we don't *have to* but we can be more relaxed about it if we're not in production during that)
19:18:46 <jeblair> that will let us serve docs from the readonly volumes which gives us HA on the backend storage
19:19:09 <fungi> once #1 is done and 400887 merges and is checked out and confirmed working, should we schedule/announce a window to do #2 (the dns change) or just trust that it will be seamless enough we can announce it agter it's done?
19:19:09 <AJaeger> anything else we want to do for that?
19:19:10 <jeblair> that gives me a warm fuzzy
19:19:12 <jeblair> (we can also, later, add HA/load balancing on the frontend webserver if needed, but that's separate)
19:19:36 <fungi> s/agter/after/
19:20:12 <AJaeger> since we can change back anytime if we notice problems, I'm fine with just doing it
19:20:13 <jeblair> fungi: i vote seamless in the general case, but coordinate with docs team so they can watch for issues)
19:20:23 <fungi> sounds fine to me
19:20:51 * annegentle catches up
19:21:25 <fungi> who from infra-root is taking point on the updated import from AJaeger's plan (item #1)?
19:21:36 <annegentle> I was going to volunteer for 3 if I understand what's needed.
19:21:56 <AJaeger> annegentle: nothing there that either of us can do
19:22:11 <clarkb> ianw had done the first pass.
19:22:18 <fungi> i think the feeling here is that none of us is entirely certain what's driving #3 in that plan
19:22:19 <annegentle> AJaeger ah, ok
19:22:41 <fungi> so want a better handle on the reasons and long-term maintenance of that archive
19:22:50 <annegentle> fungi the scenario is what I keep mentioning but can't seem to get traction on - what if someone can't find something later and comes to me because I was docs PTL at the time?
19:23:04 <AJaeger> annegentle: do we need docs-archived page or is it enough to have the archive?
19:23:27 <annegentle> AJaeger what's the technical difference? As long as my scenario means I answer "go to this thing" I'm fine.
19:23:29 <AJaeger> annegentle: http://files.openstack.org/docs-old/ has all the content, it's just not clickable
19:23:45 <fungi> yeah, we don't dispute that we'll make a copy of the old content _files_ available, just disputing the driver for having it be completely rendered and usable as a documentation site on its own
19:23:49 <annegentle> AJaeger that's probably fine
19:23:56 <AJaeger> annegentle: and we already don't have that content anymore, we redirect /bexar /cactus etc and had removed that content ages ago
19:24:19 <annegentle> fungi ok, then sounds like the retrieval use case is taken care of
19:24:32 <annegentle> AJaeger right, because we thought we'd be able to build it.
19:24:34 <annegentle> AJaeger from source
19:24:39 <annegentle> AJaeger but that's not the case now
19:24:42 <annegentle> AJaeger hence my concern
19:24:45 <AJaeger> and that should still be the case.
19:24:53 <clarkb> I can help out but don't want to take point as I am currently trying to make elasticsearch + logstash less failsauce after it went away for a week and a half over holidays
19:25:12 <AJaeger> annegentle: old versions of our tools should be able to build old documents
19:25:38 <annegentle> AJaeger oh, I thought you said on the ML that's not possible. Can't find the email on a quick look though.
19:25:47 <fungi> yeah, not impossible, just tricky to put together a platform/environment contemporary with the old content suitable for rebuilding it
19:26:30 <fungi> and not likely to be something we'd look at automating, so it would be a lot of manual effort for someone if it became necessary
19:26:45 <annegentle> fungi oh lord, no need to automate
19:26:59 <annegentle> fungi an outline that it's possible is fine to me
19:27:11 <AJaeger> it's possible, yes
19:27:25 <AJaeger> might be tricky as fungi said
19:27:46 <annegentle> ok, then my concerns are addressed. Yeah, I get the age-old problem of aging platforms. no biggie
19:28:00 * AJaeger will followup on that email thread
19:28:07 <fungi> yep, you'd probably need to get a old ubuntu lucid (or earlier?) vm booting, and install the necessary toolchain components from versions that were released around that same timeframe
19:28:09 <AJaeger> Here's what I propose now: http://paste.openstack.org/show/593810/
19:29:01 <annegentle> AJaeger with your 5) and doublechecking the rest I think it's good-to-go.
19:29:03 <AJaeger> fungi: depends on what needs building. I expect java to run on newer distros as well - and that's the main part that we need
19:29:37 <AJaeger> annegentle: let me add 6) Double check links on new docs.o.o
19:29:44 <fungi> AJaeger: that paste lgtm
19:29:50 <annegentle> AJaeger good point
19:30:35 <AJaeger> annegentle: do you mean something else with double checking?
19:30:58 <annegentle> AJaeger I meant me double-checking your paste :)
19:31:12 <annegentle> AJaeger sorry, no additional checks necessary :)
19:31:16 <AJaeger> great!
19:32:06 <AJaeger> ok, so any infra-root volunteers to help with the steps, please?
19:32:38 <fungi> ianw doesn't seem to be around for the meeting but we can try to catch up with him later in #openstack-infra and see whether he's interested in updating the production content with the missing bits and refreshing the archive volume (since he did teh earlier stages of it)
19:33:08 <jeblair> yeah, i'm not familiar with that part, so i'd love it if ianw or pabelanger could refresh it
19:33:14 <fungi> my main goal for today was to get some consensus on the remaining steps and confirmation that we can move forward with them
19:33:21 <jeblair> i'll make the read-only volumes, etc
19:33:30 <fungi> so if we don't get volunteers in-meeting that shouldn't be a show stopper
19:33:39 <fungi> i'd imagine people are still catching up from their holiday breaks
19:34:21 <clarkb> ya thats my current situation
19:34:24 <fungi> we'll circle back around after the meeting today or tomorrow and try to work out who's doing what next
19:34:34 * AJaeger will send a summary mail around to keep everybody updated
19:34:39 <fungi> any other details we need to discuss for thus?
19:34:41 <clarkb> elasticsearch/logstash went really sideways wtih us all afk so trying to correct that before moving on to other stuff
19:34:42 <fungi> s/thus/this?
19:34:50 * AJaeger is happy, thanks
19:35:20 <fungi> ugh, i need to reconnect my irc client
19:35:34 <fungi> just got a note from freenode they're about to reboot the server i'm connecting through
19:35:36 <fungi> brb
19:36:23 <RY> hello
19:37:14 <fungi> okay, i seem to be back successfully, so will continue chairing
19:37:46 <fungi> #topic Upgrading Askbot on ask.openstack.org (fungi)
19:38:02 <fungi> i think we're ready to try upgrading askbot again
19:38:08 <fungi> this is the change that will fire the upgrade process:
19:38:10 <fungi> #link https://review.openstack.org/408657 Update ask.o.o to latest 0.7.x branch r2
19:38:18 <fungi> i've marked it wip temporarily while we discuss granting mrmartin root access to the server
19:38:24 <fungi> (askbot's upgrade process is fragile and we've already had to emergency revert it once, so i feel like having hands-on troubleshooting access for him would help matters)
19:38:32 <fungi> #link https://review.openstack.org/416072 Adding user 'mkiss' to ask.o.o
19:38:40 <fungi> i'm primarily looking for infra-root consensus on that, but as always everyone is welcome to review
19:38:53 <fungi> we've also got an issue, at least on the currently deployed version, with lots of false-positive rejections from its spam filtering
19:39:00 <fungi> it has been suggested that the upgrade could solve it or at least make it easier to pin down:
19:39:01 <clarkb> oh good passes tests now
19:39:07 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2016-November/004812.html Logs for Ask.O.o - chasing false positive spam labeling
19:39:16 <fungi> that's another thing which is hard to reproduce in staging and might benefit from mrmartin having more immediate access to troubleshoot
19:42:21 <fungi> i didn't really have much else. i'm happy to drive/monitor the upgrade myself but wanted to try and get some consensus on mrmartin's shell access to it before proceeding
19:43:13 <clarkb> works for me, was waiting for tests to come back (I guess they did quickly and I just got distracted)
19:43:19 <jeblair> ++
19:44:46 <fungi> okay, cool. thanks for reviewing
19:44:55 <fungi> #topic Repo renaming (fungi)
19:45:03 <fungi> we have four repo renames proposed (in two project-config changes), so may want to consider whether it's time to schedule another rename maintenance
19:45:20 <fungi> i mostly added this to the agenda as filler in case we had time at the end (which we seem to)
19:45:48 <fungi> i know we just did a rename maintenance a few weeks ago, so if people think we should wait a little longer i'm fine with that
19:46:29 <fungi> the last one seemed to go pretty smoothly though, what with ansible automation and gerrit's online reindexing
19:46:59 <clarkb> its worth pointing out that newer gerrit has index consistency issues and online reindex problems according to gerrit ml
19:47:08 <fungi> we mainly need to be aware to give the release team a heads up, since they'd presumably want to avoid releasing anything while reindexing is underway, just to be safe
19:47:09 <clarkb> (just throwing it out there as something we should test as part of our upgrade)
19:47:20 <fungi> oh, ouch
19:47:36 <fungi> any news on whether they've pinned down the cause?
19:48:39 <zaro> clarkb: i think a fixed was just merged for that
19:49:11 <zaro> https://gerrit-review.googlesource.com/#/c/93479/
19:49:15 <clarkb> oh good
19:50:09 <fungi> so hopefully nothing we need to worry about once we upgrade, other than maybe checking for a backport
19:51:07 <fungi> given that we're pretty much all just coming back from a wasteland of feasts and booze and lack of internets, i'm inclined to push the rename discussion out and revisit during next week's meeting
19:51:39 <fungi> unless there's anyone who's eager to try and run one asap
19:52:08 <clarkb> sounds good
19:52:47 <fungi> #topic Open discussion
19:53:03 <fungi> anything else 2017 demands of us already?
19:53:12 <jlvillal> Meetbot review request: https://review.openstack.org/413222   Gives better details when doing a #undo for some of the actions.
19:53:25 <jlvillal> Low priority item...
19:54:16 <fungi> thanks jlvillal
19:54:28 <jlvillal> :)
19:54:32 <zaro> Was wondering if anybody knows where pabelanger is with the puppet fix to add zuul-launcher to openstack-ci repo?
19:54:59 <fungi> zaro: i think there was a change proposed. i'll check the discussion from before the holidays
19:56:13 <fungi> oh, he said "just doing a few tests before pushing it up"
19:56:19 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-12-20-19.03.log.html#l-10
19:57:05 <zaro> ok. Thanks.
19:57:08 <fungi> i'm not seeing it for his open reviews, so maybe he still hasn't pushed it into review.o.o yet
19:58:06 <fungi> anybody have anything else in these last two minutes?
19:59:03 <fungi> okay, i'll give you back a minute of your time ;)
19:59:06 <fungi> thanks everyone!
19:59:15 <fungi> see you all in #openstack-infra
19:59:17 <fungi> #endmeeting