19:02:05 #startmeeting infra 19:02:06 Meeting started Tue Jan 3 19:02:05 2017 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:09 The meeting name has been set to 'infra' 19:02:14 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:21 #topic Announcements 19:02:33 looks like i don't have any for this week 19:02:36 as always, feel free to hit me up with announcements you want included in future meetings 19:02:45 #topic Actions from last meeting 19:02:49 #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-12-20-19.03.html 19:02:55 "1. (none)" 19:02:59 good work! 19:03:03 O/ 19:03:05 i have to admit, i did a lot of that 19:03:11 not ashamed 19:03:28 #topic Specs approval 19:03:35 we don't seem to have anything new up this week 19:03:55 #topic Priority Efforts: Docs Publishing via AFS (AJaeger) 19:03:58 #link http://lists.openstack.org/pipermail/openstack-docs/2016-December/009430.html production docs.o.o cut-over plan 19:04:08 we were talking about this in irc yesterday, and it feels like we're ready to forge ahead here 19:04:29 the plan looks pretty solid 19:05:01 i was just catching up on that this morning 19:05:08 yes, I think we're ready... 19:05:35 i had one question -- why make docs-archive.o.o? is files.o.o not sufficient for being able to retrieve a file in the (unlikely) event that we need it? 19:05:36 updated plan at http://lists.openstack.org/pipermail/openstack-docs/2017-January/009436.html 19:06:00 jeblair: you cannot really follow links 19:06:27 all links to / will go to files.o.o instead of to the docs root 19:06:32 okay, but i thought the archive was so that we can pull a file out and copy it over if we find it's missing 19:06:32 AJaeger: are they absolute, not relative? 19:07:08 it really depends - and some have docs.o.o hardcoded 19:07:13 creating a new vhost to serve obsolete content seems, at best, unecessary, and at worst, actually counter productive 19:08:06 i guess the primary need is to have a copy of the files in case we forgot anything after we've told rackspace to tear down the cloudsites account. i'm less concerned about giving it a dedicated vhost and custom 404 forwarding but those also sound pretty cheap to add 19:09:13 it does also seem like we should stop serving a dedicated site for the stale archive after some period of time, even if we do continue to keep copies of the files around 19:09:52 mainly to avoid confusion like jeblair mentions 19:09:54 okay. it's a significant change to the plan. i'd love it if we had a good reason for it, and a plan for retiring it, etc. but i'm not going to veto it if someone wants to do that (extra) work. 19:11:05 i'm mostly ambivalent, though curious to see arguments for why the files.o.o interface is really insufficient 19:11:07 we can move forward without the extra vhost - and discuss the need of a vhost with the rest of the docs team. 19:11:14 o/ 19:11:21 (like, to me, the fact that links don't work is not a problem as i never envisioned the purpose of this to be that we would say "oh, you're looking for that doc? it's hosted on docs-archive.openstack.org" 19:11:48 i thought the purpose was to say "oh, crap, we're missing a doc, let me find it in the archive and manually copy it over") 19:12:31 my intention is indeed manually copying it over 19:12:58 * AJaeger checks some old emails 19:13:05 if you make docs-archive.o.o, make sure you make a fully-exclude robots.txt 19:13:38 right, we really really really don't want that appearing in search engines 19:13:54 indeed 19:15:04 okay, so are we at least agreed we can move forward with steps 1 and 2 for now, and discuss the necessity of step 3 with the docs team? 19:15:36 yes 19:15:42 yes for 1... 19:16:04 what other blockers do we have for 2? 19:16:04 for 2 -- i pushed up a patch to do a vos release in cron so that we could serve docs from a readonly volume 19:16:12 oh, that hasn't merged yet? 19:16:16 i'm not sure if it landed 19:16:24 pretty sure i reviewed it, but checking 19:16:27 AJaeger: i think you were fixing it up? 19:16:46 #link https://review.openstack.org/400887 19:16:46 * AJaeger checks 19:16:51 i seem to have new-year amnesia so i don't trust my pre-holiday memories at this point 19:17:00 yeah, working through some fog myself :) 19:17:10 jeblair: yes, I fixed it 19:17:19 we may want to land that and create the replica volumes before doing #2 19:17:23 okay, looks like i did review it 19:17:49 * AJaeger updates plan and adds that change 19:18:17 (we don't *have to* but we can be more relaxed about it if we're not in production during that) 19:18:46 that will let us serve docs from the readonly volumes which gives us HA on the backend storage 19:19:09 once #1 is done and 400887 merges and is checked out and confirmed working, should we schedule/announce a window to do #2 (the dns change) or just trust that it will be seamless enough we can announce it agter it's done? 19:19:09 anything else we want to do for that? 19:19:10 that gives me a warm fuzzy 19:19:12 (we can also, later, add HA/load balancing on the frontend webserver if needed, but that's separate) 19:19:36 s/agter/after/ 19:20:12 since we can change back anytime if we notice problems, I'm fine with just doing it 19:20:13 fungi: i vote seamless in the general case, but coordinate with docs team so they can watch for issues) 19:20:23 sounds fine to me 19:20:51 * annegentle catches up 19:21:25 who from infra-root is taking point on the updated import from AJaeger's plan (item #1)? 19:21:36 I was going to volunteer for 3 if I understand what's needed. 19:21:56 annegentle: nothing there that either of us can do 19:22:11 ianw had done the first pass. 19:22:18 i think the feeling here is that none of us is entirely certain what's driving #3 in that plan 19:22:19 AJaeger ah, ok 19:22:41 so want a better handle on the reasons and long-term maintenance of that archive 19:22:50 fungi the scenario is what I keep mentioning but can't seem to get traction on - what if someone can't find something later and comes to me because I was docs PTL at the time? 19:23:04 annegentle: do we need docs-archived page or is it enough to have the archive? 19:23:27 AJaeger what's the technical difference? As long as my scenario means I answer "go to this thing" I'm fine. 19:23:29 annegentle: http://files.openstack.org/docs-old/ has all the content, it's just not clickable 19:23:45 yeah, we don't dispute that we'll make a copy of the old content _files_ available, just disputing the driver for having it be completely rendered and usable as a documentation site on its own 19:23:49 AJaeger that's probably fine 19:23:56 annegentle: and we already don't have that content anymore, we redirect /bexar /cactus etc and had removed that content ages ago 19:24:19 fungi ok, then sounds like the retrieval use case is taken care of 19:24:32 AJaeger right, because we thought we'd be able to build it. 19:24:34 AJaeger from source 19:24:39 AJaeger but that's not the case now 19:24:42 AJaeger hence my concern 19:24:45 and that should still be the case. 19:24:53 I can help out but don't want to take point as I am currently trying to make elasticsearch + logstash less failsauce after it went away for a week and a half over holidays 19:25:12 annegentle: old versions of our tools should be able to build old documents 19:25:38 AJaeger oh, I thought you said on the ML that's not possible. Can't find the email on a quick look though. 19:25:47 yeah, not impossible, just tricky to put together a platform/environment contemporary with the old content suitable for rebuilding it 19:26:30 and not likely to be something we'd look at automating, so it would be a lot of manual effort for someone if it became necessary 19:26:45 fungi oh lord, no need to automate 19:26:59 fungi an outline that it's possible is fine to me 19:27:11 it's possible, yes 19:27:25 might be tricky as fungi said 19:27:46 ok, then my concerns are addressed. Yeah, I get the age-old problem of aging platforms. no biggie 19:28:00 * AJaeger will followup on that email thread 19:28:07 yep, you'd probably need to get a old ubuntu lucid (or earlier?) vm booting, and install the necessary toolchain components from versions that were released around that same timeframe 19:28:09 Here's what I propose now: http://paste.openstack.org/show/593810/ 19:29:01 AJaeger with your 5) and doublechecking the rest I think it's good-to-go. 19:29:03 fungi: depends on what needs building. I expect java to run on newer distros as well - and that's the main part that we need 19:29:37 annegentle: let me add 6) Double check links on new docs.o.o 19:29:44 AJaeger: that paste lgtm 19:29:50 AJaeger good point 19:30:35 annegentle: do you mean something else with double checking? 19:30:58 AJaeger I meant me double-checking your paste :) 19:31:12 AJaeger sorry, no additional checks necessary :) 19:31:16 great! 19:32:06 ok, so any infra-root volunteers to help with the steps, please? 19:32:38 ianw doesn't seem to be around for the meeting but we can try to catch up with him later in #openstack-infra and see whether he's interested in updating the production content with the missing bits and refreshing the archive volume (since he did teh earlier stages of it) 19:33:08 yeah, i'm not familiar with that part, so i'd love it if ianw or pabelanger could refresh it 19:33:14 my main goal for today was to get some consensus on the remaining steps and confirmation that we can move forward with them 19:33:21 i'll make the read-only volumes, etc 19:33:30 so if we don't get volunteers in-meeting that shouldn't be a show stopper 19:33:39 i'd imagine people are still catching up from their holiday breaks 19:34:21 ya thats my current situation 19:34:24 we'll circle back around after the meeting today or tomorrow and try to work out who's doing what next 19:34:34 * AJaeger will send a summary mail around to keep everybody updated 19:34:39 any other details we need to discuss for thus? 19:34:41 elasticsearch/logstash went really sideways wtih us all afk so trying to correct that before moving on to other stuff 19:34:42 s/thus/this? 19:34:50 * AJaeger is happy, thanks 19:35:20 ugh, i need to reconnect my irc client 19:35:34 just got a note from freenode they're about to reboot the server i'm connecting through 19:35:36 brb 19:36:23 hello 19:37:14 okay, i seem to be back successfully, so will continue chairing 19:37:46 #topic Upgrading Askbot on ask.openstack.org (fungi) 19:38:02 i think we're ready to try upgrading askbot again 19:38:08 this is the change that will fire the upgrade process: 19:38:10 #link https://review.openstack.org/408657 Update ask.o.o to latest 0.7.x branch r2 19:38:18 i've marked it wip temporarily while we discuss granting mrmartin root access to the server 19:38:24 (askbot's upgrade process is fragile and we've already had to emergency revert it once, so i feel like having hands-on troubleshooting access for him would help matters) 19:38:32 #link https://review.openstack.org/416072 Adding user 'mkiss' to ask.o.o 19:38:40 i'm primarily looking for infra-root consensus on that, but as always everyone is welcome to review 19:38:53 we've also got an issue, at least on the currently deployed version, with lots of false-positive rejections from its spam filtering 19:39:00 it has been suggested that the upgrade could solve it or at least make it easier to pin down: 19:39:01 oh good passes tests now 19:39:07 #link http://lists.openstack.org/pipermail/openstack-infra/2016-November/004812.html Logs for Ask.O.o - chasing false positive spam labeling 19:39:16 that's another thing which is hard to reproduce in staging and might benefit from mrmartin having more immediate access to troubleshoot 19:42:21 i didn't really have much else. i'm happy to drive/monitor the upgrade myself but wanted to try and get some consensus on mrmartin's shell access to it before proceeding 19:43:13 works for me, was waiting for tests to come back (I guess they did quickly and I just got distracted) 19:43:19 ++ 19:44:46 okay, cool. thanks for reviewing 19:44:55 #topic Repo renaming (fungi) 19:45:03 we have four repo renames proposed (in two project-config changes), so may want to consider whether it's time to schedule another rename maintenance 19:45:20 i mostly added this to the agenda as filler in case we had time at the end (which we seem to) 19:45:48 i know we just did a rename maintenance a few weeks ago, so if people think we should wait a little longer i'm fine with that 19:46:29 the last one seemed to go pretty smoothly though, what with ansible automation and gerrit's online reindexing 19:46:59 its worth pointing out that newer gerrit has index consistency issues and online reindex problems according to gerrit ml 19:47:08 we mainly need to be aware to give the release team a heads up, since they'd presumably want to avoid releasing anything while reindexing is underway, just to be safe 19:47:09 (just throwing it out there as something we should test as part of our upgrade) 19:47:20 oh, ouch 19:47:36 any news on whether they've pinned down the cause? 19:48:39 clarkb: i think a fixed was just merged for that 19:49:11 https://gerrit-review.googlesource.com/#/c/93479/ 19:49:15 oh good 19:50:09 so hopefully nothing we need to worry about once we upgrade, other than maybe checking for a backport 19:51:07 given that we're pretty much all just coming back from a wasteland of feasts and booze and lack of internets, i'm inclined to push the rename discussion out and revisit during next week's meeting 19:51:39 unless there's anyone who's eager to try and run one asap 19:52:08 sounds good 19:52:47 #topic Open discussion 19:53:03 anything else 2017 demands of us already? 19:53:12 Meetbot review request: https://review.openstack.org/413222 Gives better details when doing a #undo for some of the actions. 19:53:25 Low priority item... 19:54:16 thanks jlvillal 19:54:28 :) 19:54:32 Was wondering if anybody knows where pabelanger is with the puppet fix to add zuul-launcher to openstack-ci repo? 19:54:59 zaro: i think there was a change proposed. i'll check the discussion from before the holidays 19:56:13 oh, he said "just doing a few tests before pushing it up" 19:56:19 #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-12-20-19.03.log.html#l-10 19:57:05 ok. Thanks. 19:57:08 i'm not seeing it for his open reviews, so maybe he still hasn't pushed it into review.o.o yet 19:58:06 anybody have anything else in these last two minutes? 19:59:03 okay, i'll give you back a minute of your time ;) 19:59:06 thanks everyone! 19:59:15 see you all in #openstack-infra 19:59:17 #endmeeting