19:01:28 <clarkb> #startmeeting infra
19:01:29 <openstack> Meeting started Tue Aug  7 19:01:28 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:32 <openstack> The meeting name has been set to 'infra'
19:02:20 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:32 <clarkb> #topic Announcements
19:02:49 <clarkb> hrm I guess bot isn't update topics right now (is it not oper?)
19:03:34 <fungi> hopefully that helps
19:03:44 <clarkb> PTG is coming up. If we have time at end of meeting we might brainstorm agenda a bit
19:03:49 <fungi> [i already had to do that once last week]
19:04:24 <fungi> [[might be something is breaking +O mode]]
19:04:30 <clarkb> I'll also put out an ethercalc link to figure out when is a good night to have biergarten trip in denver
19:04:38 <clarkb> fungi: possibly related to +r?
19:04:53 <fungi> your guess is as good as mein
19:05:06 <ianw> fungi: hrm, just rolled out changes to accessbot to mlock, that runs infrequently, i wonder if related
19:05:27 <clarkb> #topic Actions from last meeting
19:05:41 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-07-31-19.01.txt Minutes from last meeting
19:06:37 <clarkb> No explicit actions recorded, but mordred has been hard at work getting ansible things going, cmurphy's puppet-4 stack is almost completely merged, and ianw's spec for letsencrypt has been updated
19:06:42 <clarkb> Thank you everyone for pushing on that stuff
19:06:54 <clarkb> #topic Specs approval
19:07:24 <clarkb> ianw: how are we feeling about https://review.openstack.org/#/c/563849/ now?
19:08:34 * fungi has partially read, saw there was a new revision today
19:08:35 <clarkb> This is the 3rd party ci spec. I seem to recall we wanted to hold on it a bit, but not sure if that happened between now and last meeting or if that was last meeting
19:08:39 <ianw> oh, i owed an update to that to focus on windmill, which i have failed to do
19:08:43 <clarkb> ah ok
19:08:44 <fungi> oh, wait, i was thinking of another spec
19:09:42 <clarkb> in that case I won't propose we put it up for approval this week :)
19:09:51 <fungi> missed that one too while on vacation but starred to catch up on
19:09:56 <clarkb> fungi: thanks
19:10:03 <clarkb> #topic Priority Efforts
19:10:15 <clarkb> fungi: want to start with storyboard?
19:10:25 <fungi> oh, hrm
19:10:46 * diablo_rojo_phon lurks
19:10:52 <clarkb> looks like search changes are happening?
19:11:10 <diablo_rojo_phon> Yep :) fatema has been working on that
19:11:11 <fungi> name-based project urls are finally working on storyboard.o.o
19:11:20 <diablo_rojo_phon> Her internship is getting close to done though.
19:11:25 <fungi> #link https://storyboard.openstack.org/#!/project/openstack-infra/storyboard Name-based project URLS!
19:11:41 <diablo_rojo_phon> I thiiink she'll keep working on stuff but when school starts it will probably slow a bit.
19:11:50 <diablo_rojo_phon> Woot woot!
19:11:58 <fungi> there's a lengthy stack of webclient improvements SotK has up for review
19:12:00 <clarkb> neat to both things. The name based urls will make a lot of people happy
19:12:11 <diablo_rojo_phon> Yes it will.
19:12:12 <corvus> i've seen changes to start updating readmes!  very exciting!
19:12:13 <clarkb> and I'm sure improved search will be welcomed by many as well
19:12:36 <fungi> yeah, especially the ability to sort stories by status in search results
19:12:47 <corvus> i'll celebrate by approving one of them now
19:12:59 <clarkb> corvus: such productive celebration
19:13:04 <fungi> (and there was much rejoicing)
19:13:05 <diablo_rojo_phon> Awesome :) thanks corvus
19:13:53 <diablo_rojo_phon> There are definitely a lot of patches with my +2 that just need one more
19:14:06 <diablo_rojo_phon> Trying to get on top of all of that a bit more.
19:14:07 <fungi> last week we performed our first project-group rename in sb
19:14:23 <fungi> (it was a trivial update query changing a single row)
19:14:32 <diablo_rojo_phon> But still :)
19:14:34 <clarkb> fungi: this was the api-sig change?
19:15:10 <fungi> yup. they has a project-group but thankfully project-groups (like most resources in sb) have integer ids and the names are just a mapping table
19:15:13 <clarkb> ( I didn't realize that was the first case of that particular rename)
19:15:52 <fungi> we've renamed lots of projects in sb but no project-groups before now
19:16:52 <clarkb> other than needing reviews anything else before we move on?
19:17:33 <fungi> nope, not that i can think of
19:17:44 <fungi> i probably missed some stuff while i was travelling though
19:17:55 <clarkb> Alright #2 priority effort is the config mgmt modernization.
19:18:08 <cmurphy> o/
19:18:13 <fungi> i did see some of that going on but also wasn't able to follow super closely yet
19:18:16 <clarkb> I approved the change to make this an official priority effort. We had mostly operated under that assumption anyway
19:18:37 <clarkb> The big change in the last week is we now have a node bootstrapped by ansible only and is a replacement bastion host
19:19:07 * fungi will love to see the old puppetmaster instance finally go away
19:19:09 <clarkb> it is called bridge.openstack.org. The idea here was we needed a more modern bastion since trusty python is getting unhappy particularly with cryptography
19:19:16 <clarkb> and we need to uprade anyway
19:19:29 <clarkb> Eventually it will take over the triggering of puppet on remote nodes but it is not doing that quite yet
19:19:34 <fungi> what was the story behind the name choice there? i'm sure there was one
19:19:47 <mordred> clark suggested it and I just ran with it :)
19:19:51 <fungi> it's a bridge from our old model to eventually just using zuul?
19:19:54 <clarkb> fungi: monty asked for name ideas and I threw out a few and said I liked bridge the most
19:20:00 <clarkb> fungi: more star trek bridge
19:20:10 <mordred> yah. it's "the bridge"
19:20:11 <fungi> aha
19:20:11 <clarkb> but it fits that way too :)
19:20:32 <corvus> patrick stewart is apparently going to reprise jean-luc in a new show.  this is relevant information for this meeting.
19:20:34 <mordred> I could use more reviews on that stack - I'm a little bit stalled at the moment needing to get the install_modules change landed
19:20:38 * fungi musters to his station
19:20:39 <ianw> should have changed the motd to "make it so"
19:20:41 <clarkb> the big thing to be aware of at the moment is we need to double account hiera data changes on puppetmaster.o.o and brdige.o.o until bridge takes over the puppet triggering
19:20:47 <mordred> and yes, that
19:21:08 <mordred> I don't think we're too far away from being able to move triggering to bridge and decommission puppetmaster
19:21:12 <clarkb> corvus: cautiously optomistic about that :)
19:21:16 <corvus> also, you *are* allowed to call it "bilge" when things go wrong
19:21:28 <mordred> oh - also, double-accounting any password info in /root too
19:21:49 <corvus> mordred: maybe we could decide to just use bridge for root/passwords.gpg?
19:21:56 <corvus> since that's just manual
19:22:06 <clarkb> corvus: that is probably a good idea
19:22:08 <mordred> yah. I'm fine with that
19:22:17 <clarkb> maybe send a followup to the fyi email thread saying we are doing ^
19:22:29 <corvus> mordred: can you do one final sync, then move the old ones out of the way and replace with a README?
19:22:34 <mordred> maybe let's set u-w on passwords.gpg on puppetmaster just to be sure ... o rthat
19:22:36 <mordred> yes
19:22:42 <corvus> ya, something like that :)
19:22:49 * mordred will do that
19:23:24 <clarkb> mordred: I'll also rebuild the rax dns venv on bridge so that ssl works
19:23:38 <clarkb> (I hadn't done that on puppet master yet)
19:24:04 <clarkb> Semi related are the set of changes at topic:puppet-4
19:24:20 <clarkb> I think I'm mostly up to date on my reviews there if someone else is willing tomake a followup pass through
19:24:30 <clarkb> but the vast majority of those changes have been merged at this point
19:24:44 <cmurphy> i think we're stuck on ask and ask-staging
19:24:55 <cmurphy> since ask-staging seems to not exist right now
19:24:59 <clarkb> oh right, ianw ^ is there a tl;dr for the status on ask-staging?
19:25:11 <clarkb> (I think you had mentioned that there were some things)
19:25:32 <ianw> oh, hrm ... so i think i was waiting on a change to get numerical hosts for it into puppet
19:25:41 <ianw> and then probably got distracted
19:26:08 <ianw> yes, https://review.openstack.org/#/c/558992/
19:26:46 <clarkb> ianw: ya I left a couple comments on that one last week when I was trying to get the future parser stuff going
19:26:58 <clarkb> ianw: I think we need to set up the hiera groups in puppet for that (and then add hiera data)
19:27:14 <clarkb> cmurphy: would it make sense to skip ask* for now and continue with another service or three?
19:27:35 <cmurphy> clarkb: sure
19:27:46 <ianw> i can do that, we really should look at getting ask.o.o updated and will need the staging host
19:27:57 <clarkb> ianw: ++
19:28:05 <ianw> there's an email out there i sent when i investigated issues with ask.o.o
19:28:12 <clarkb> I think we can do the puppet 4 parser work in parallel (on other services) and update ask
19:28:15 <cmurphy> i've just been going through site.pp looking for candidates
19:28:40 <cmurphy> i have a change to remove stackalytics.o.o since that seesm to not exist anymore https://review.openstack.org/585208
19:28:42 <clarkb> cmurphy: ethercalc, ehterpad, logstash-workers, elasticsearch, paste are favorites of mine fwiw. They tend to be straightforward
19:28:56 <cmurphy> clarkb: ether* scare me because nodejs
19:29:24 <fungi> storyboard-dev?
19:29:42 <cmurphy> also there are a few in-flight beaker tests for ether* that i think are still unmerged
19:30:20 <clarkb> cmurphy: we have etherpad-dev too but getting tests happy first is probably a good idea
19:30:20 <cmurphy> fungi: storyboard-dev is good except it's slightly less dev than other -dev sites since some people are using it to trial storyboard
19:30:35 <cmurphy> also i noticed that odsreg.o.o seems very sad
19:30:50 <fungi> i thought we removed that
19:31:06 <clarkb> I want to say we did
19:31:12 <fungi> i need to follow up with ttx to find out if i missed deleting that after he said i could or something
19:31:29 <cmurphy> it's still in site.pp
19:31:42 <cmurphy> if it's not needed i'll propose a change to remove it
19:31:45 <clarkb> and dns
19:31:50 <clarkb> we should clean that up if possible
19:31:56 <fungi> i'll take care of it
19:31:59 <clarkb> thanks
19:32:07 <fungi> after i check that it's really gone from the tenant too
19:33:03 <clarkb> mordred: cmurphy: anything else we should be aware of with the changes that are happening?
19:33:30 <cmurphy> nothing from me i think
19:34:10 <fungi> yeah, not finding odsreg in our tenant, so it's just puppet/dns i need to clean up i guess
19:34:35 <clarkb> #topic General Topics
19:34:44 <clarkb> corvus: Swift logs is back on the agenda
19:35:36 <corvus> we're very close to being able to switch
19:35:54 <corvus> we need to land this change:
19:36:03 <corvus> #link https://review.openstack.org/588383 Add severity info to logstash and filter out DEBUG lines
19:36:13 <corvus> to the log processor
19:36:24 <corvus> that should be compatible with the current system too, so is safe to land now
19:36:27 <clarkb> I've reviewed it and has my +2. It should be forward and backward compatible
19:36:29 <clarkb> yup
19:36:58 <corvus> then i can push up some test changes (to, say, devstack) so we can see how everything will look
19:37:05 <corvus> then we should be able to switch
19:37:21 <corvus> so, basically, now's the time to start talking about whether we want to, any concerns folks have, etc.
19:37:34 <clarkb> corvus: we should clarify to openstack release team that this won't affect the releaes artifacts (all of that goes to tarballs still aiui)
19:37:50 <clarkb> I'm surethey will be concered about it impacting the release (but its logs only shold be quite safe)
19:38:05 <clarkb> corvus: is the plan to use vexxhost to start?
19:38:45 <corvus> currently, it's only easy to use one provider.  we can use vexxhost or ovh.  i think we should be able to randomly pick one of the two in the near future if we want, but i haven't implemented that yet.
19:39:10 <corvus> we *might* be able to extend it to support rax's not-swift.  no promises.
19:39:35 <corvus> those are the only things vaguely like swift that i believe we have access to at the moment.
19:40:35 <mordred> ++
19:40:57 <corvus> there are definite downsides to this.  severity filtering in the ui will be gone.  we'll be storing plain text *and* htmlified logs.  and tristanC apparently wanted to use os-loganalyze for some of the machine learning log stuff.
19:41:19 <clarkb> we are trading complexity for reliability and retention
19:41:33 <clarkb> I'm personally happy to use grep myself if I have to in return for having logs for longer period of time
19:42:02 <corvus> that's how i look at it.  and i'd like to look into implementing some of that complexity in javascript if we can.
19:42:31 <clarkb> vim users, you can open http(s) urls directly in vim too. Quite useful for this stuff
19:42:36 <clarkb> I'm sure emacs does similar
19:42:44 <fungi> yeah, i expect some css/js magic would make it possible to filter in-browser (though the browser would still have to retrieve the full log first)
19:43:32 <corvus> if we can't, maybe we should revisit the idea of a proxy.  tobiash is running one anyway (for access control purposes), so we might want to revisit that.  but at the moment at least, i feel like i'd like to try to avoid that.  it's where we hit the wall last time we tried swift.
19:43:48 <fungi> and the implementation gives us a durable url to the index for any given build?
19:44:17 <corvus> fungi: yeah.  https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/logs_78/587178/1/check/tox-py35-on-zuul/1f46ccf/  is the latest test
19:44:21 <clarkb> fungi: until the expiration happens and it is deleted. Couple that with zuul dashboard and you should be able to find builds reasonably well
19:44:51 <corvus> oh, yeah, icons are still to-do.
19:44:58 <mordred> I think it's a reasonable tradeoff
19:45:00 <fungi> meh, icons ;)
19:45:10 <mordred> especially given the cost to th eproject overall whenver logs.o.o goes south
19:45:18 <clarkb> mordred: ya
19:45:54 <clarkb> we have 15 minutes left, ok if we move on? Please go review the log processor change if you have some cycles
19:46:00 <mordred> and I agree, we can always run a proxy in the future - OR we could just run a web service that does os-loganalyze/machine-learning like things that is not what we normally route people to for log viewing
19:46:06 <corvus> yep.
19:46:17 <corvus> sounds like folks want to move forward still.
19:46:20 <mordred> clarkb: I already +A'd the processor change
19:46:24 <clarkb> mordred: woot
19:46:27 <corvus> let me know if you have concerns.  or, you know, propose patches to fix them.  :)
19:46:55 <fungi> do we want a redirector of some sort (rather than a proxy) in case we need to move objects to another container or provider or something? or just plan to do phased transitions and consider the loss of that data an acceptable risk in such situations?
19:47:19 <corvus> fungi: personally, i'd suggest never moving data
19:47:27 <corvus> if we lose a provider holding logs, we lose those logs
19:48:08 <clarkb> not all that different from the 30 day expiry on mega fs today
19:48:12 <fungi> since we don't control those domains though, we have to assume domain name changes will never occur, and urls paths will never be rejiggered, right?
19:48:26 <clarkb> fungi: yes, though I think that is sort of the point of those services.
19:48:33 <fungi> wfm
19:48:35 <corvus> the nice thing about not having a proxy (at least, in the current system) is that we no longer have a single hostname for logs.  each log has its own url.  a job that runs today may be stored in vexxhost.  tomorrow in ovh.
19:48:38 <clarkb> fungi: I would be surprised if they change those ever
19:48:55 <corvus> so us changing log-storageprovider
19:48:57 <corvus> gah
19:49:02 <corvus> so us changing log-storage-providers is a non-event
19:49:05 <fungi> yep, i'm fine with all that as long as we set expectations around it
19:49:40 <fungi> still no guarantee of any particular retention in cases where we have to move off of a provider we were using
19:49:48 <mordred> yup
19:49:54 <fungi> (thinking ahead to a broadly distributed object storage future for logging)
19:50:13 <mordred> mmm. broadly distributed
19:50:22 <corvus> fungi: yeah, we may want to not enable log storage on a provider until we have a reasonable expectation they'll stick around for 6 months :)
19:50:29 <fungi> heh
19:50:32 <fungi> good call!
19:50:48 <clarkb> Really quickly, the winterscale naming thread hsa an update from jbryce with a proposal. Would appreciate all yall's feedback. I think we are also keeping it to that private thread we are on for now while some last details are sorted.
19:51:21 <clarkb> feedback that has been provided so far is positive though. I am excited :)
19:51:26 <mordred> ++
19:51:34 <corvus> yeah, hopefully we can make that public again soon, but it's good to know there's progress
19:51:39 * mordred is excited - and also excited to get to the point where we can talk about it publically
19:51:44 <clarkb> ++
19:52:03 <clarkb> #topic Open Discussion
19:52:57 <clarkb> Couple items from me that weren't quite proper agenda worthy: I am taking tomorrow off to go fishing and spend time with family. I expect that I will be properly afk. The other thing is PTG planning is quickly bubbling up to the top of my todo list
19:53:16 <clarkb> I think the config management stuff is getting to the point where we should hav a good idea of what we can dig into in denver
19:53:19 <clarkb> which is good
19:55:02 <corvus> https://zuul-ci.org/users.html  and  http://superuser.openstack.org/articles/zuul-case-study-bmw/  are cool
19:55:40 <tobiash> :)
19:55:49 <fungi> as is https://www.podcastinit.com/zuul-with-monty-taylor-episode-172/
19:55:51 <corvus> well, except for that none of the links go to zuul-ci.org.  i'll email them i guess :)
19:56:11 <clarkb> corvus: dhellmann and I have been asked to start working on similar from an openstack perspective
19:56:38 <clarkb> happy to inclue others if they are interested (maybe to review?)
19:57:05 <corvus> cool!
19:57:57 <fungi> eek, yeah i see one nodepool link gong to our openstack deployment operational docs, and another going to a github mirror
19:58:26 <clarkb> Alright we are basically at time and I have a leftover burger patty to turn into lunch
19:58:33 <clarkb> Thank you everyone
19:58:53 <fungi> thanks clarkb!
19:58:58 <clarkb> #endmeeting