19:02:38 <fungi> #startmeeting infra
19:02:38 <openstack> Meeting started Tue Nov 15 19:02:38 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:42 <openstack> The meeting name has been set to 'infra'
19:02:59 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:03 <fungi> #topic Announcements
19:03:09 <fungi> #info The http://developer.openstack.org/ site is now served from an AFS-backed Apache server under Infra team management.
19:03:17 <zara_the_lemur__> yay!
19:03:17 <fungi> congrats all who worked on that! i'm sure docs.o.o is coming up soon
19:03:29 <fungi> #info REMINDER: Our "Ocata Cycle" [2048R/0x8B1B03FD54E2AC07] signing key has been generated; infra-root admins are requested to follow our attestation process as soon as possible.
19:03:37 <fungi> #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xd47bab1b7dc2e262a4f6171e8b1b03fd54e2ac07&fingerprint=on OpenStack Infra (Ocata Cycle) <infra-root@openstack.org>
19:03:43 <fungi> #link http://docs.openstack.org/infra/system-config/signing.html#attestation Attestation process
19:03:45 <fungi> thanks pabelanger and pleia2 for doing this already!
19:03:54 <fungi> i'd like to see at least a few more infra-root signatures on this before we switch it in for the old newton signing key
19:03:57 <fungi> (which is set to expire in just over two weeks!)
19:04:04 <fungi> as always, feel free to hit me up with announcements you want included in future meetings
19:04:08 * clarkb makes note to get on board and do that
19:04:37 * jhesketh apologises for not having done it yet and will do it today
19:04:42 <fungi> no worries, and thanks
19:04:45 <fungi> #topic Actions from last meeting
19:04:52 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-11-08-19.03.html last meeting's minutes
19:05:01 <fungi> fungi send summit session summary to infra ml
19:05:08 <fungi> still in progress but nearly done :/
19:05:13 <fungi> #action fungi send summit session summary to infra ml
19:05:24 <fungi> pabelanger send followup announcement for wheel-less pep8 job transition
19:05:26 <fungi> #link http://lists.openstack.org/pipermail/openstack-dev/2016-November/106990.html "important changes to pep8 python jobs" reminder
19:05:29 <fungi> thanks again pabelanger--i'm looking forward to the transition tomorrow!
19:05:36 <fungi> #action pabelanger un-wip "Force gate-{name}-pep8-{node} to build needed wheels" change
19:05:54 <fungi> and then we'll merge it early-ish in the day i guess?
19:06:07 <fungi> #link https://review.openstack.org/391875 "Force gate-{name}-pep8-{node} to build needed wheels" change
19:06:18 <fungi> fungi propose clarification about project-specific test preconditions (databases, filesystems) for cti
19:06:22 <fungi> #link https://review.openstack.org/397502 "Acknowledge nominal prerequisites for tests" change
19:06:28 <fungi> it's wip, but figured i'd get people to have a look real quick and see if that matches what we agreed to in the last meeting before i get it onto the tc agenda for next week
19:08:12 <fungi> but i guess you can always look at it later
19:08:36 <fungi> fungi purchase cert for pholio.openstack.org
19:08:42 <fungi> #link https://review.openstack.org/397464 "Use CA-issued SSL cert for pholio.o.o service" change
19:08:48 <fungi> all set up in hiera and ready to go
19:08:53 <fungi> just needs approval
19:09:01 <fungi> and that's it for the action items i think
19:09:08 <fungi> #topic Specs approval
19:09:26 <fungi> #info APPROVED: "Automate Creating Branches" spec
19:09:41 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/branch-automation.html
19:10:17 <fungi> zaro also has a spec proposed for "Gerrit 2.13 Upgrade" which he plans to bring up next week (he's unavailable this week)
19:10:41 <fungi> #link https://review.openstack.org/388190 "Gerrit 2.13 Upgrade" spec
19:10:49 <pabelanger> o/
19:10:50 <fungi> he's also suggesting it as a priority spec
19:11:10 <fungi> #topic Priority Efforts
19:11:28 <fungi> nothing is called out on the agenda, though as i said a moment ago, next week we may vote on adding another one
19:11:58 <fungi> #topic Priority Efforts: Zuul v3
19:12:14 <fungi> just a quick mention that the zuul subteam had their first meeting yesterday
19:12:39 <fungi> #link http://eavesdrop.openstack.org/#Zuul_Meeting A weekly meeting to discuss Zuul development
19:12:45 <fungi> for those who want to start attending
19:13:10 <fungi> #topic Zanata Upgrade to Xenial (eumel8)
19:13:18 <fungi> #link https://blueprints.launchpad.net/openstack-i18n/+spec/ocata-zanata-upgrade%7C
19:13:45 <fungi> i can't actually browse that, eumel8
19:13:50 <eumel8> yes, thx. I18n team wants to upgrade zanata server. We're looking for volunteers :-)
19:14:05 <clarkb> basiaclly newer zanata for newer features requires newer java which requires newer ubuntu
19:14:15 <eumel8> upgrade means change to java8 with change to xenial VM
19:14:18 <fungi> looks like the %7c on the end is an error
19:14:22 <fungi> #undo
19:14:22 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x7fac552e9850>
19:14:25 <zara_the_lemur__> (the link seems to work sans the %7C)
19:14:26 <zara_the_lemur__> oh snap
19:14:42 <fungi> #link https://blueprints.launchpad.net/openstack-i18n/+spec/ocata-zanata-upgrade "Ocata: Zanata upgrade" I18n blueprint
19:14:43 <eumel8> starting on translate-dev, test the stuff and then the live upgrade
19:15:23 <fungi> is the current puppet module set up such that we can easily deploy different versions of zanata on translate-dev.o.o vs translate.o.o?
19:15:41 <clarkb> fungi: it should be, I merged a change recently to be smarter about choosing the java version
19:15:57 <clarkb> and assuming there are no other unforseen issues that should be all that is required to run on xenial -dev and trusty prod
19:16:00 <eumel8> yes, it should be ready
19:16:02 <pabelanger> have we done a xenial server yet? I remember pleia2 asking for it a while back
19:16:24 <fungi> so we can just start with a proposed change to the zanata release for it, and then we rebuild the server on a xenial image?
19:16:45 <fungi> do we need to disable ansible runs for te existing translate-dev.o.o i guess?
19:17:01 <clarkb> pabelanger: no we haven't done any xenial servers yet, this will likely be the first
19:17:12 <fungi> does it keep its own state locally, or is it all in trove and cinder?
19:17:15 <pabelanger> clarkb: okay
19:17:32 <clarkb> fungi: I don't know about cinder, but pretty sure it uses trove. At least for dev I don't know that we care about the state though
19:17:48 <fungi> well, we've done other servers running xenial, but i don't think we've upgraded any existing services from trusty to xenial yet
19:18:13 <pabelanger> might be worth picking a server in our inventory, and upgrading it to xenial first. Confirm puppet is working as expected before adding new puppet manifests
19:18:15 <clarkb> (probably good for testing to try and preserve state though)
19:18:31 <clarkb> pabelanger: yes I think we are saying zanata-dev is that server :)
19:18:35 <fungi> pabelanger: well, as i said, we do have some puppeted servers running xenial
19:18:48 <clarkb> and this isn't a new puppet manifest, its an existing one just reapplied to newer distro release
19:18:53 <pabelanger> fungi: oh, sorry. I missed that
19:18:57 <pabelanger> okay, cool
19:19:15 <fungi> for example, firehose01.o.o is xenial
19:19:36 <pabelanger> nice, wasn't aware of that
19:19:43 <fungi> we've just only been using xenial for new services so far and haven't moved any existing services from trusty to xenial as far as i'm aware
19:20:07 <fungi> but it seems relatively low-risk, especially for the -dev server
19:20:31 <pabelanger> agreed
19:20:39 <clarkb> eumel8: was it you that fixed the puppet to handle different java versions?
19:20:45 <fungi> need to make sure that we also update the node entry for that server in the global site manifest to be tested against xenial instead of trusty
19:20:49 <clarkb> in any case i18n team has been pretty proactive doing things like ^ which is great too
19:20:53 <eumel8> clarkb: yes
19:21:28 <fungi> so seems likely to be fairly straightforward. just need an infra-root volunteer to try replacing teh server
19:21:47 <clarkb> ya I had mentioned at the summit I could probably do that but then havebeen swamped with post summit catching up of all the things
19:21:58 <pabelanger> I can also help if needed
19:22:02 <clarkb> I can probably give it a go late this week if no one else is able to
19:22:11 <clarkb> probably thursday or friday
19:22:32 <fungi> #action clarkb pabelanger work on replacing translate-dev.openstack.org with xenial server running newer zanata release
19:22:35 <fungi> thanks!
19:22:46 <fungi> eumel8: anything else you wanted to cover on this?
19:22:53 <eumel8> thanks!
19:23:03 <eumel8> no, thanks, fungi
19:23:27 <fungi> thanks for reminding us the i18n team is ready for newer zanata
19:23:38 <fungi> #topic Scheduling another project rename maintenance (fungi)
19:23:39 <eumel8> :)
19:23:57 <fungi> we've had a proposed project rename sitting out there for about 6 weeks
19:24:24 <fungi> #link https://review.openstack.org/381332 'Rename abbreviation name "rsc" (rack scale controller) to final name "valence".' change
19:25:06 <fungi> i was expecting us to accumulate a few more and batch them up, but it looks like it's just this one
19:25:25 <AJaeger> yeah, nothing else on the radar so far
19:25:31 <fungi> anybody up for a (quick) project rename maintenance soon?
19:25:53 <fungi> online reindexing worked well last time, so we don't need those massive outages any longer it seems
19:26:33 <clarkb> next week is major holiday week if I have done my maths right
19:26:39 <clarkb> so likely needs to happen this week or the week after
19:26:45 <fungi> just a brief window to make the filesystem/db/group changes and rename in gituhb
19:27:00 <clarkb> (or maybe enough people don't observe said major holiday and don't mind doing it while the rest of us eat too much turkey)
19:28:02 <fungi> yeah, my calendar confirms thanksgiving holiday in the usa is next week
19:28:40 <ianw> well, i could have a go, but ... i've never done it before, and those that have might not be around
19:28:53 <clarkb> if it happens monday or tuesday I can likely be around
19:29:41 <fungi> i don't have much on my calendar for friday, and with it being a short maintenance i could see having a fairly short advance notification period for that if we send something to teh -dev ml today
19:30:23 <fungi> i'd say announce it as a 30-minute window though expect the outage to be less than 15
19:31:06 <clarkb> (I will be beach bound on friday, I get to experience fungi's daily life for a weekend)
19:31:27 <fungi> oh! have fun
19:32:18 <fungi> i could see doing the maintenance on monday i guess
19:32:35 <fungi> anybody happen to know the author of that change's irc nick?
19:32:54 <mordred> o/
19:32:59 <fungi> just wondering how best to go giving their contributors a heads up that it's changing
19:33:02 <mordred> sorry for being late
19:33:14 <fungi> mordred: welcome back to internets
19:33:22 <clarkb> fungi: lin_yang maybe
19:33:24 <clarkb> ?
19:33:35 <clarkb> (is in -infra)
19:34:35 <fungi> yeah, just asked in there
19:34:59 <fungi> okay, so we could say monday tentative... 22:00 utc?
19:35:20 <fungi> earlier? later?
19:35:31 <ianw> isn't that zuul meeting?
19:35:34 <clarkb> looking at my calendar I can do before 1900UTC
19:35:37 <fungi> oh, yep!
19:35:47 <clarkb> otherwise its dentist appointments (ugh hate when those sneak up on you)
19:35:53 <fungi> how about 18:00 utc monday then?
19:35:57 <clarkb> that I can do
19:36:10 <fungi> i was trying to go a little later in case ianw wanted to help
19:36:22 <clarkb> oh ya, well you also don't need me
19:36:27 <clarkb> or we could do tuesday after our meeting?
19:36:35 <clarkb> ~2100UTC?
19:36:53 <fungi> i expect to be in the tc meeting at that point
19:36:59 <lin_yang> Thanks a lot to all of you to make rename rsc->valence happen. really appreciate
19:36:59 <fungi> but i could do 22:00
19:37:03 <clarkb> tc is from 2000-2100 isn't it?
19:37:08 <fungi> oh!
19:37:17 <fungi> yep, 21:00 utc totally works for me
19:37:20 <fungi> ;)
19:37:25 <clarkb> ianw: ^ that work?
19:37:37 <ianw> yep, i'll be around
19:37:58 <fungi> lin_yang: if we rename it on tuesday (a week from today) will that be okay with you and your contributors?
19:38:02 <ianw> learn the ropes from the old sea dogs :)
19:38:28 <lin_yang> fungi: that's okay
19:38:38 <fungi> excellent
19:39:04 <clarkb> fungi: sorry I had already offset "after our meeting" to accomodate tc meeting but didn't make that clear :)
19:39:26 <fungi> #info Gerrit will be offline for roughly 15 minutes between 21:00 and 21:30 UTC on Tuesday, November 22 for a project rename maintenance.
19:39:40 <fungi> does that look right?
19:39:51 <fungi> anybody want to send an announcement to the -dev ml?
19:40:03 <clarkb> that looks correct to me
19:40:29 <ianw> i can send an announcement today
19:41:03 <fungi> thanks ianw!
19:41:21 <fungi> #action ianw send maintenance announcement for project rename
19:41:53 <fungi> i think we're out of topics now, other than...
19:41:54 <fungi> #topic Open discussion
19:42:39 <clarkb> fungi: on my change to document test envs, you didn't -1 but provided a lot of feedback. Do you want that feedback addressed before merging or merge soonish and address the feedback in followups/
19:43:03 <jeblair> clarkb: oh, what's the link for that change?
19:43:10 <fungi> clarkb: i'm fine either way. was mostly waiting for more feedback to collect on it before either you updated it or i voted
19:43:12 <clarkb> jeblair: https://review.openstack.org/#/c/394566/1
19:43:16 <clarkb> fungi: gotcha
19:43:18 <ianw> clarkb: so what's the deal with image uploads?  we know for sure of issues with rax until next shade release, right?
19:43:30 <fungi> i'm also fine if you don't address everything in there, or ask me to submit an update
19:43:36 <clarkb> ianw: correct, we need to restart builder today to pick up new shade (was released earlier today I think)
19:43:43 <clarkb> ianw: that will fix the setting of metadata on the images in rax
19:43:44 <jeblair> yes, was released earlier today
19:43:53 <ianw> clarkb: ok, i can do that later today my time when things are quiet
19:44:02 <clarkb> ianw: cool
19:44:28 <jeblair> 1.13.1
19:44:43 <zara_the_lemur__> noting to everyone that starting tomorrow, the StoryBoard meeting is moving from 15:00 UTC 19:00UTC (same day and channel, ie: Weds in #openstack-meeting); maybe I should've put this on the agenda.
19:44:52 <mordred> ianw, clarkb: do we want to finish landing ianw's logging changes for the restart?
19:44:57 <pabelanger> now that https://review.openstack.org/#/c/395056/ has landed (RETRY_LIMIT patch). When do we want to schedule a restart of zuul?
19:44:58 <fungi> oh, great reminder. thanks zara_the_lemur__!
19:45:10 <clarkb> mordred: ianw I think those got in? or did I misread things last night?
19:45:14 <ianw> mordred: already landed them yesterday, so we're getting some split out logs /var/log/nodepool/upload
19:45:20 <mordred> woot!
19:45:26 <jeblair> pabelanger: you could just do a hard-restart of all the launchers when the system isn't too busy
19:45:31 <jeblair> that's what i've been doing
19:45:56 <jeblair> oh wait
19:45:59 <jeblair> the launchers don't need to restart
19:46:01 <jeblair> it's the scheduler
19:46:01 <zara_the_lemur__> fungi: np. :) we'll also make sure to post something to the list in case anyone misses it here.
19:46:02 <pabelanger> will that work?
19:46:07 <pabelanger> Ya
19:46:59 <jeblair> pabelanger: then i'd just save the check/gate pipelines and then restart when it's not too busy (and there's nothing in the release pipeline)
19:47:17 <fungi> #info The StoryBoard meeting time is moving 4 hours later (to 19:00 UTC Wednesdays in #openstack-meeting) starting this week.
19:47:19 <pabelanger> okay, maybe I'll do that for tomorrow morning. Getting late in the day here
19:48:13 <AJaeger> pabelanger: looks rather idle now - but one change in release queue
19:48:24 <AJaeger> (release-post)
19:48:46 <fungi> pabelanger: make sure to give the release team a heads up (in #openstack-release is generally fine) before the zuul restart so they know not to approve anything until you're done
19:48:53 <pabelanger> AJaeger: my only concern is I won't be around to support the restart if we go now
19:49:05 <pabelanger> fungi: roger!
19:49:10 <AJaeger> pabelanger: ok, understood now
19:49:37 <jeblair> pabelanger: (also, as a special case, if the only jobs running in the release pipeline are mirror rebuilds, it's okay to restart)
19:51:35 <fungi> right, those are idempotent and will just run again soon thereafter anyway
19:51:57 <fungi> so no actual loss of state by aborting them
19:53:41 <ianw> pabelanger: want me to do it my afternoon?
19:53:59 <fungi> zuul is a ghost town during ianw's afternoon
19:54:08 <pabelanger> ianw: sure, if you have the time
19:54:23 <fungi> would probably be a good time, plus pabelanger would probably be around soon thereafter to monitor it
19:54:35 <pabelanger> I mean, we can do it now, if people are around to help support.  I just have to run out for some family things shortly
19:55:07 <ianw> ok, i'll save the queues and do it about 2pm my time if that's what we want
19:55:13 <fungi> i don't think we're in a huge hurry. just a couple of changes stuck in check right now loping jobs
19:55:18 <fungi> er, looping
19:55:32 <jeblair> they're l(o)oping along
19:55:33 <AJaeger> down to one - the other got abandoned
19:56:10 <fungi> ahh, well, would still be good to make sure that remaining one gets reenqueued and then keep an eye on it to make sure it end with a MAX_RETRIES state
19:56:23 <fungi> er, make sure it endS
19:56:40 <clarkb> if we want to stop it in the interim you can log into the host running the job and kill thigns until the job fails
19:56:44 <jeblair> RETRY_LIMIT i think is the word we used
19:56:56 <jeblair> clarkb: it'll be a good test case :)
19:56:57 <AJaeger> change 389654 is the one
19:57:08 <clarkb> jeblair: yup
19:57:24 <AJaeger> but 397756 is running for 6 hours, so might be a canditate as well ;(
19:57:57 <fungi> er, right-o. i blanked on the status name
19:59:23 <ianw> so was the conclusion to restart it this afternoon, and watch that 389654 completes?
19:59:46 <fungi> ianw: in _your_ "this afternoon" i think, yes
19:59:58 <fungi> which would be many of our "early tomorrow morning"
20:00:19 <fungi> oh, we're out of time--thanks everyone!
20:00:20 <AJaeger> yes, check that 389654 completes - there's a job getting restarted again and again, with the new change it will only be started 3 times.
20:00:25 <fungi> #endmeeting