19:01:04 <clarkb> #startmeeting infra
19:01:05 <openstack> Meeting started Tue Sep  3 19:01:04 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:08 <openstack> The meeting name has been set to 'infra'
19:01:17 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-September/006474.html
19:01:43 <clarkb> #topic Announcements
19:01:53 <dmsimard> clarkb: added a last minute topic on the wiki.
19:02:01 <AJaeger_> o/
19:02:03 <clarkb> dmsimard: ok if we have time we can get to it
19:02:12 <dmsimard> sure
19:02:22 <clarkb> We are in openstack TC and PTL election period. If you would like to run for a position you have about ~4 hours left to nominate yourself
19:02:47 <AJaeger_> any nominations for Infra yet?
19:02:53 <clarkb> I've nominated myself for Infra PTL. As noted in that change If I do get elected for the U cycle I'd like to plan for this to be my last cycle
19:03:02 <AJaeger_> thanks, clarkb !
19:03:03 <clarkb> AJaeger_: at least check just the one (myself)
19:03:44 <clarkb> Given that I'd like to put effort into mentoring more team members and leaders in the group if there is interest
19:03:55 <clarkb> please feel free to reach out if you have any interest in being more involved
19:04:09 <clarkb> Other announcements: fungi I suppose we shouldn't count on your presence much the next few days?
19:05:07 <fungi> probably not, no
19:05:13 <fungi> i'll check in when i can
19:05:39 <clarkb> fungi: let us know if you need help with stuff that is time sensitive
19:06:36 <clarkb> #topic Actions from last meeting
19:06:42 <fungi> will do, but i'm mostly avoiding time-sensitive stuff ;)
19:06:53 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-08-27-19.01.txt minutes from last meeting
19:06:58 <clarkb> fungi: good idea :)
19:07:03 <clarkb> I see no actions from last meeting
19:07:37 <clarkb> #topic Priority Efforts
19:07:42 <clarkb> #topic OpenDev
19:08:02 <clarkb> Yesterday we noticed that 4 of 8 gitea backends had stale gitconfig lockfiles which prevented gerrit git replication from happening
19:08:07 <clarkb> thank you ianw for cleaning that up
19:08:37 <clarkb> I think ianw tracked that back to poor IO performance based on journald complaints in dmesg at roughly the same time
19:08:55 <clarkb> ianw: ^ anything to add to that?
19:09:11 <clarkb> I've still not come up with a great way to work around that. Maybe we need to update gitea to clean up stale locks?
19:09:28 <ianw> i just had a thought that maybe we should put some randomisation into the git gc cron job, i think they all kick off at the same time?
19:09:52 <clarkb> they do kick off at the same time
19:11:04 <fungi> yeah, maybe staggering them will help avoid a thundering herd on the storage backend
19:11:23 <clarkb> seems like a reasonable thing to try
19:11:39 <AJaeger> good idea
19:12:32 <clarkb> ianw: do you want to write that chnage or should we find a volunteer?
19:12:39 <ianw> clarkb: just doing it now :)
19:12:43 <clarkb> great thanks
19:12:51 <clarkb> Any other opendev business before we move on?
19:14:16 <clarkb> #topic Update Config Management
19:14:39 <clarkb> We merged a bunch of changes from mordred to build gerrit images
19:15:24 <Shrews> clarkb: he also put up this one: https://review.opendev.org/678428
19:15:48 <clarkb> #link https://review.opendev.org/#/q/status:open+(topic:gus2019+OR+topic:gerrit-images) Gerrit and docker related changes that need review
19:16:23 <clarkb> Shrews: cool that should be captured by the query above
19:16:29 <Shrews> yah
19:16:43 <clarkb> Would be great if people can take some time to get through those as I think interesting and useful things were learned at the gerrit user summit
19:18:28 <Shrews> any reason we haven't +Ad https://review.opendev.org/678413 yet?
19:18:57 <clarkb> Shrews: fungi notes that it will need someone to restart gerrit
19:19:04 <clarkb> I can probably work on that tomorrow
19:19:49 <fungi> yeah, alternatively it could just happen as part of the rename maintenance?
19:21:49 <clarkb> probably best to make sure we don't break thing as ahead of time?
19:22:13 <clarkb> just to reduce potential impacts during renames. I can likely work on that this week (today is weird as have school stuff with kids this afternoon)
19:23:01 <clarkb> Anything else? if not we'll move on to storyboard
19:23:26 <diablo_rojo_phon> Nothing currently.
19:24:08 <clarkb> #topic Storyboard
19:24:14 <clarkb> diablo_rojo_phon: says nothing to report :)
19:24:31 <diablo_rojo_phon> I do.
19:24:35 <fungi> i did get mysql slow query logging reestablished and whip up a refreshed analysis
19:24:45 <diablo_rojo_phon> Oh cool.
19:24:46 <clarkb> oh that was for last topic then? in any case go for it on storyboard
19:24:57 <fungi> #link http://files.openstack.org/user/fungi/storyboard/
19:25:17 <fungi> mentioned last week in #storyboard
19:25:36 <fungi> now that i've worked out the process, i'm happy to do further updates for it we can compare
19:25:57 <diablo_rojo_phon> fungi: I suppose I should commence asking mordred to take a look?
19:26:42 <fungi> SotK also mentioned he might try to tease out some useful nuggets for ideas of what to improve
19:28:03 <clarkb> diablo_rojo_phon: he is currently on vacation but imagine as soon as he is back bugging him would be great :)
19:29:06 <clarkb> Anything else on storyboard?
19:29:43 <diablo_rojo_phon> Will do!
19:29:48 <diablo_rojo_phon> No I don't think so.
19:29:57 <clarkb> #topic General Topics
19:30:07 <clarkb> #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup
19:30:08 <dmsimard> Need to pick up kids from school, if I'm not back we can talk about my topic in the infra channel
19:30:14 <clarkb> dmsimard: ok
19:30:18 <clarkb> first up is server upgrades
19:30:25 <clarkb> fungi: ^ anything new to add re wiki?
19:30:51 <fungi> not really, no
19:31:13 <clarkb> Removing/replacing static.o.o is next
19:31:19 <clarkb> #link https://etherpad.openstack.org/p/static-services
19:31:25 <fungi> other community responsibilities have taken priority this past week. i hope to pick it back up soon (but welcome anyone who wants to help out with it too)
19:31:27 <clarkb> ianw: ^ I tried to capture a todo list at the bottom of that document
19:31:49 <clarkb> ianw: can you check if it is accurate and we can start to ask for volunteers?
19:32:03 <ianw> oh sorry, yeah forgot to do that, will do
19:32:14 <clarkb> thanks
19:32:17 <clarkb> thanks
19:32:22 <clarkb> silly keyboard
19:32:57 <clarkb> Next up is AFS mirrors and debugging failures to vos release tem
19:32:59 <clarkb> *them
19:33:15 <clarkb> ianw: auristor mentioned rsync may be to blame (and possibly the switch from rsync on xenial to bionic?))
19:33:39 <clarkb> have we learned anything concrete yet and/or are there more changes for us to review?
19:34:13 <ianw> i think fedora has locked itself again
19:34:35 <ianw> the reason is takes forever to release now is still unclear ...
19:34:54 <ianw> looking at the rysnc "-i" output, it does not do something like touch every single file
19:35:00 <fungi> you ruled out metadata being modified i guess
19:35:11 <ianw> but maybe updating the metadata on the upper directories is enough
19:35:26 <ianw> to determine this i think we need to turn on file auditing on the servers again
19:36:26 <ianw> #link https://lists.openafs.org/pipermail/openafs-info/2019-August/042861.html
19:37:25 <ianw> the only follow up was basically that version mis-match of client/server shouldn't matter ... that really only leaves bionic rsync v xenial rsync i guess :/
19:37:47 <clarkb> at least we could rule out the version mismatch
19:38:05 <ianw> i will keep poking so we can get it stable
19:38:20 <clarkb> thanks
19:38:43 <clarkb> Next up is pointing out the current plan to do project renames on September 16
19:39:03 <clarkb> I expect we'll do that about 7am Pacific time to get as early a start as possible?
19:39:13 <clarkb> (assuming I'm helping could be earlier if others want to drive it)
19:39:39 <clarkb> If you know of projects that need to be renamed they are highly encouraged to do that during this round of renames :)
19:40:29 <AJaeger> we should sent out an announcement...
19:40:46 <clarkb> AJaeger: good idea. I'm guessing your day is about done. I'll do that after the meeting
19:41:36 <AJaeger> thanks, clarkb
19:42:09 <clarkb> Next is PTG planning. I've been unable to spend time on this yet, but wanted to point out that it isn't too early for anyone to suggest topics
19:42:16 <clarkb> #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 Feel free to add PTG topics
19:43:31 <clarkb> That was it for the agenda that I sent out yesterday. dmsimard are you back to talk about ara and swift hosted logs?
19:43:48 <clarkb> if not we can pick up the conversation in the infra channel as you mentioned /me gives it a few minutes before openning the floor
19:46:31 <fungi> we also tend to be pretty efficient at coming up with things to talk about or hack on at the ptg once we get into a room together
19:46:58 <clarkb> #topic Open Discussion
19:47:06 <fungi> i like to treat the planning list as more of a resource to help jog our memories so we don't forget things we wanted to talk about
19:47:10 <clarkb> We'll talk ara in the infra channel. Anything else before we call it a meeting
19:47:24 <fungi> nothing on my end
19:47:27 <clarkb> fungi: ++ ya its more about having good notes so that we can let the room move without spending a ton of time looking info up
19:47:41 <fungi> agreed
19:51:30 <clarkb> sounds like that is it
19:51:35 <clarkb> thank you everyone. We'll end a bit early today
19:51:37 <clarkb> #endmeeting