19:01:50 <clarkb> #startmeeting infra
19:01:51 <openstack> Meeting started Tue Oct 10 19:01:50 2017 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:55 <openstack> The meeting name has been set to 'infra'
19:02:06 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:07 <cmurphy> o/
19:02:23 <clarkb> #topic Announcements
19:02:25 <fungi> ooh, meetins
19:02:41 <clarkb> we'll talk about this more later but
19:02:44 <clarkb> #link http://lists.openstack.org/pipermail/openstack-dev/2017-October/123337.html Zuulv3 Back to Production
19:02:50 * mordred waves
19:03:03 <clarkb> Zuulv3 will be going back to managing the gate tomorrow
19:03:32 <clarkb> The other item I wanted ot call out is that TC nominations are happening now
19:03:50 <clarkb> They end at 23:45UTC today (whihc is about 5 hours and 40 minutes from now
19:04:35 <clarkb> if you intend on running don't forget to get nominated. Also good luck to those of you already running
19:04:54 <clarkb> #topic Actions from last meeting
19:05:07 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-10-03-19.01.txt Minutes from last meeting
19:05:28 <clarkb> last meeting was quite a bit more informal. Just wanted to make sure we weren't letting other important items slip while we were preoccupied with zuul
19:05:53 <clarkb> #topic Specs approval
19:06:15 <clarkb> #link https://review.openstack.org/#/c/509257/ mark gerrit upgrade as complete
19:06:35 <clarkb> I plan to merge ^ later today if there aren't any objections.
19:06:57 <clarkb> Other than that I think we've mostly been swamped with zuul related things the last couple weeks which takes us to
19:07:03 <clarkb> #topic Priority Efforts
19:07:11 <clarkb> #topic Zuul v3
19:07:58 <clarkb> as mentioned earlier the plan is to put this back in production tomorrow as our gate keeper
19:08:13 <jeblair> i've been cleaning up:
19:08:17 <jeblair> #link https://etherpad.openstack.org/p/zuulv3-issues
19:08:28 <jeblair> so that we can continue to use it to help firefighting tomorrow
19:08:40 <fungi> much cleaner!
19:08:48 <jeblair> it has pretty clear sections now -- triage, job issues, zuul issues
19:08:55 <jeblair> also fixed issues
19:09:10 <jeblair> our list of fixed issues from the past week is *REALLY LONG*  yay!
19:09:16 <jeblair> so i moved it to
19:09:20 <jeblair> #link https://etherpad.openstack.org/p/zuulv3-fixed-issues
19:09:41 <jeblair> basically, i think it's good to keep fixed issues on the main etherpad for a little while, for communication purposes
19:10:12 <jeblair> but we didn't need all those there, so that's a place they can retire to
19:10:45 <jeblair> i plan to restart zuul with a bunch of fixes soon
19:10:53 <jeblair> i think those are the main things for folks to know?
19:10:53 <mordred> \o/
19:11:05 <mordred> yah. things seem WAY better now
19:11:18 <clarkb> Earlier this morning there seemed to be ocnfusion around what projects should/can be doing nowish to prepare
19:11:26 <jeblair> i'll roll the repartition into the restart too
19:11:29 <clarkb> might be good to send another email nowish explaining the general procss for not infra?
19:13:16 <mordred> clarkb: agree - I can take a stab at writing that if that's helpful
19:13:23 <clarkb> mordred: that would be great
19:14:33 <clarkb> jeblair: and I guess just need reviewers to keep going through that list of reviews at the top of the etherpad?
19:15:21 <jeblair> clarkb: yeah; looks like a lot of job changes showed up recently
19:15:29 <jeblair> so there's currently more job work than system work
19:16:03 <fungi> i do expect we were sending some mixed messages on whether zuul -1 votes were worth looking into because of the previously semi-prevalent post_failures
19:16:41 <jeblair> yeah.  we also plan to have some more post_failures before tomorrow as well (to help track down a bug)
19:17:04 <jeblair> i think the message should be something like "all zuul -1 errors should at least be triaged at this point"
19:17:50 <fungi> and i'll admit the backlog for normal check in v3 right now is making it hard for people to know whether the previous failures are still relevant or just stale data for things we've already fixed, because rechecks are taking over a day to process
19:18:12 <clarkb> ya, unfortunately we had to pull a lcoud out of the rotation too so that likely is only going to get worse
19:18:25 <jeblair> it'll get "better" in maybe 1 hour when i drop the queues
19:18:31 <mordred> yah
19:18:34 <jeblair> then, of course, start to get worse again
19:18:46 <fungi> until around noon utc tomorrow
19:18:53 <jeblair> then hopefully better again :)
19:19:00 <ianw> heh, yeah my browser couldn't load the status page yesterday
19:19:06 <jeblair> but at least between now and then, votes will be a little more current
19:19:19 <Shrews> request list has more than doubled since yesterday, so it's not getting better w/o some intervention
19:19:20 <fungi> ianw: my browser couldn't load the status page days ago, so you must have one heck of a browser
19:19:28 <jeblair> mordred: maybe include a note about the fresh restart in the email?
19:20:19 <jeblair> 7340 corvus    20   0 2042744 427196  64988 R 100.0  1.3   3203:13 chromium-browse
19:20:28 <jeblair> that's my dedicated status page browser
19:20:34 <jeblair> (i use ff for real work)
19:20:42 <fungi> wow, nuts
19:21:00 <jeblair> also, is unusually warm in here
19:21:06 <fungi> are you actually able to scroll the page that way?
19:21:21 <jeblair> not well
19:21:47 <clarkb> how big is the check queue now? eg is it so much larger than we expect it to be nomrally?
19:21:55 <clarkb> looks like ~1100
19:22:13 <clarkb> and generally we have ~300 changes in status page in total if we ignore merge-check (we aren't using it in v3)
19:22:16 <fungi> and then there's periodic
19:22:20 <jeblair> plus 467 in periodic
19:22:48 <jeblair> probably tripleo-check too
19:23:02 <clarkb> but ya firefox 57 beta almost manages to keep up fwiw
19:23:14 <jeblair> 121 in check-tripleo
19:23:49 <mordred> jeblair: yah (re fresh restart)
19:23:53 <fungi> oh, also if i'm interpreting the discussion on #-dev correctly it sounds like the dedicated tripleo pipelines might be able to go away in another week-ish? that will simplify the config somewhat
19:24:04 <mordred> ooh. that'd be nice
19:24:04 <jeblair> so, we'll manage through the cutover.  status page improvements to come after cutover.
19:24:20 <clarkb> might be worth explaining ^ in the update email too
19:24:36 <clarkb> basically status page is in an exceptional state now and not expected to stay that way post cut over
19:24:53 <jeblair> ++
19:25:13 <fungi> yeah, i have a feeling lots of people think the status page is entirely unusable given experiences with attempting to render the extreme backlog status data
19:25:31 <fungi> when in actuality it'll probably be semi-usable under normal peak load
19:27:32 <clarkb> ok, anything else zuul related, especially cut over related?
19:27:55 <jeblair> nak
19:28:17 <fungi> just... you know... excited!
19:28:26 <clarkb> #topic General Topics
19:28:32 <fungi> and also going to bed early tonight
19:28:49 <clarkb> I had one item for this, the new contributor portal project needs to be created
19:28:58 <clarkb> Do we want to wait for post cutover to worry about that?
19:29:10 <clarkb> otherwise we'll need to think carefully about running it in both versions of zuul
19:29:19 <clarkb> #link https://review.openstack.org/#/c/509943/
19:30:16 <clarkb> thingee: if you are around, are you able to wait a couple days while we do the next zuulv3 rollout?
19:30:31 <fungi> sounded like they're sort of pushing to announce it rsn
19:33:16 <clarkb> Looks like AJaeger has +2'd, I'll need to follow up with thingee and see what timeline they are operating under as  Ithink my preference would be to merge the change after the v3 rerollout
19:33:42 <clarkb> but it has configs for both versions so maybe we just merge it now as is?
19:33:52 <clarkb> anyone have concerns with ^
19:34:15 <fungi> i don't object
19:34:45 <fungi> he's done the work to set it up for v2 and v3, so i don't think we have a good reason to hold it up
19:34:51 <jeblair> yeah, let's go for it
19:34:54 <clarkb> ok
19:35:00 <clarkb> I'll give it a review post meeting
19:35:15 <clarkb> #topic Project Renames
19:35:25 <clarkb> I think we are actually ready to think about doing these again :)
19:35:34 <fungi> wow, it's been a while!
19:35:44 <clarkb> #link https://review.openstack.org/#/c/500768
19:35:50 <clarkb> we have at least one rename in the pipeline
19:36:35 <clarkb> This week is probably bad as is week before summit travel, but there is one week between, 16th-20th where we could potentially do some renames
19:37:30 <clarkb> do we want to provisionally plan for the 20th, then commit to it in a week at our next meeting if we've caught up with everything else?
19:37:41 <fungi> i don't have any atypical obligations that week, so count me in
19:38:27 <fungi> i usually try to catch the release team meeting 15:00-16:00 on fridays, but otherwise nothing specific planned
19:38:39 <clarkb> I've got a dentist visit and a user group talk that week on the 17th and 19th but should be able to help out on the 20th
19:39:35 <clarkb> ok lets aim for the 20th and actually commit to it next meeting if everything else is calmer :)
19:39:51 <clarkb> say at 1600UTC ish
19:39:52 <fungi> wfm
19:40:33 <clarkb> #info plan for project renames around 1600UTC October 20 assuming we have cauhgt up on other items by then. Decide if we are ready for that in the next meeting
19:40:41 <clarkb> #topic General Discussion
19:40:55 <clarkb> #undo
19:40:56 <openstack> Removing item from minutes: #topic General Discussion
19:41:04 <clarkb> #topic Open Discussion
19:41:08 <ianw> fungi: you mentioned last time wanting a hand with writing a project update?
19:42:34 <clarkb> Was that the summit project update?
19:42:38 <fungi> ianw: well, wanting some volunteers to help the ptl do that anyway (which i'm also willing to help with)
19:42:57 <clarkb> ya, I think my rough plan for that was to dig in after the zuul re roll out
19:42:58 <mordred> how does this: https://etherpad.openstack.org/p/OCPH0jEDn9 look for email about what people shold do?
19:43:00 <fungi> and possibly also appearing on stage with said ptl
19:43:09 <clarkb> as I think that will largely affect the content :)
19:43:23 <clarkb> if you are interested in helping with that let me know and we can start drafting it up together
19:43:38 <clarkb> (I think fungi and paul have already expressed interestg in helping, but hte more the merryier)
19:44:02 <ianw> ok, etherpad?
19:44:16 <clarkb> ianw: ya I think we'll start on an etherpad
19:44:38 <clarkb> #link https://etherpad.openstack.org/infra-sydney-summit-project-update <- how about that one
19:45:14 <jeblair> mordred: lgtm, added one note under "A few notes about tomorrow"
19:45:34 <ianw> clarkb: excellent, open in a tab right next to zuulv3-issues :)
19:48:00 <clarkb> Oh I've also signed us up for an onboarding session that we will share with tonyb in sydney
19:48:12 <fungi> mordred: lgtm
19:48:25 <clarkb> I'll likely be working off of what fungi put together for Boston in that session. If you want to help with that too let me know
19:48:57 <clarkb> mordred: might be good to add something about please talk to us
19:49:16 <clarkb> mordred: I think the projects that we would consider to be successful in all the transition stuff have been willing to work and talk with us directly
19:49:42 <mordred> clarkb: good point - adding
19:51:35 * clarkb gives it a couple more minutes for any last items but then everyone can go find $meal early :)
19:52:47 <jlvillal> Review request: https://review.openstack.org/#/c/509670/  openstack-infra/os-loganalyze  Support the log level "CRITICAL"
19:53:23 <jlvillal> As it is now. If you select ERROR, then you will not see CRITICAL messages.
19:53:25 <mordred> clarkb: how's that?
19:53:37 <jlvillal> Or if you basically select any log level you won't see CRITICAL messages.
19:54:03 <clarkb> mordred: I like that, really hte important part is communicating to help make sure we don't miss important stuff
19:54:03 <mordred> jlvillal: oh wow - when did CRITICAL get added?
19:54:13 <clarkb> mordred: it is apparnetly part of python logging
19:54:25 <jlvillal> mordred: Uh, not sure. It is part of the python logging standard.
19:55:23 <clarkb> ok, thanks everyone. You canfind us in #openstack-infra or on the infra mailing list. Please do let us know if you have questions about the v3 rollout or see weird unexpected behavior
19:55:26 <clarkb> and with that
19:55:28 <clarkb> #endmeeting