19:01:28 #startmeeting infra 19:01:29 Meeting started Tue Aug 7 19:01:28 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:30 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:32 The meeting name has been set to 'infra' 19:02:20 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:32 #topic Announcements 19:02:49 hrm I guess bot isn't update topics right now (is it not oper?) 19:03:34 hopefully that helps 19:03:44 PTG is coming up. If we have time at end of meeting we might brainstorm agenda a bit 19:03:49 [i already had to do that once last week] 19:04:24 [[might be something is breaking +O mode]] 19:04:30 I'll also put out an ethercalc link to figure out when is a good night to have biergarten trip in denver 19:04:38 fungi: possibly related to +r? 19:04:53 your guess is as good as mein 19:05:06 fungi: hrm, just rolled out changes to accessbot to mlock, that runs infrequently, i wonder if related 19:05:27 #topic Actions from last meeting 19:05:41 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-07-31-19.01.txt Minutes from last meeting 19:06:37 No explicit actions recorded, but mordred has been hard at work getting ansible things going, cmurphy's puppet-4 stack is almost completely merged, and ianw's spec for letsencrypt has been updated 19:06:42 Thank you everyone for pushing on that stuff 19:06:54 #topic Specs approval 19:07:24 ianw: how are we feeling about https://review.openstack.org/#/c/563849/ now? 19:08:34 * fungi has partially read, saw there was a new revision today 19:08:35 This is the 3rd party ci spec. I seem to recall we wanted to hold on it a bit, but not sure if that happened between now and last meeting or if that was last meeting 19:08:39 oh, i owed an update to that to focus on windmill, which i have failed to do 19:08:43 ah ok 19:08:44 oh, wait, i was thinking of another spec 19:09:42 in that case I won't propose we put it up for approval this week :) 19:09:51 missed that one too while on vacation but starred to catch up on 19:09:56 fungi: thanks 19:10:03 #topic Priority Efforts 19:10:15 fungi: want to start with storyboard? 19:10:25 oh, hrm 19:10:46 * diablo_rojo_phon lurks 19:10:52 looks like search changes are happening? 19:11:10 Yep :) fatema has been working on that 19:11:11 name-based project urls are finally working on storyboard.o.o 19:11:20 Her internship is getting close to done though. 19:11:25 #link https://storyboard.openstack.org/#!/project/openstack-infra/storyboard Name-based project URLS! 19:11:41 I thiiink she'll keep working on stuff but when school starts it will probably slow a bit. 19:11:50 Woot woot! 19:11:58 there's a lengthy stack of webclient improvements SotK has up for review 19:12:00 neat to both things. The name based urls will make a lot of people happy 19:12:11 Yes it will. 19:12:12 i've seen changes to start updating readmes! very exciting! 19:12:13 and I'm sure improved search will be welcomed by many as well 19:12:36 yeah, especially the ability to sort stories by status in search results 19:12:47 i'll celebrate by approving one of them now 19:12:59 corvus: such productive celebration 19:13:04 (and there was much rejoicing) 19:13:05 Awesome :) thanks corvus 19:13:53 There are definitely a lot of patches with my +2 that just need one more 19:14:06 Trying to get on top of all of that a bit more. 19:14:07 last week we performed our first project-group rename in sb 19:14:23 (it was a trivial update query changing a single row) 19:14:32 But still :) 19:14:34 fungi: this was the api-sig change? 19:15:10 yup. they has a project-group but thankfully project-groups (like most resources in sb) have integer ids and the names are just a mapping table 19:15:13 ( I didn't realize that was the first case of that particular rename) 19:15:52 we've renamed lots of projects in sb but no project-groups before now 19:16:52 other than needing reviews anything else before we move on? 19:17:33 nope, not that i can think of 19:17:44 i probably missed some stuff while i was travelling though 19:17:55 Alright #2 priority effort is the config mgmt modernization. 19:18:08 o/ 19:18:13 i did see some of that going on but also wasn't able to follow super closely yet 19:18:16 I approved the change to make this an official priority effort. We had mostly operated under that assumption anyway 19:18:37 The big change in the last week is we now have a node bootstrapped by ansible only and is a replacement bastion host 19:19:07 * fungi will love to see the old puppetmaster instance finally go away 19:19:09 it is called bridge.openstack.org. The idea here was we needed a more modern bastion since trusty python is getting unhappy particularly with cryptography 19:19:16 and we need to uprade anyway 19:19:29 Eventually it will take over the triggering of puppet on remote nodes but it is not doing that quite yet 19:19:34 what was the story behind the name choice there? i'm sure there was one 19:19:47 clark suggested it and I just ran with it :) 19:19:51 it's a bridge from our old model to eventually just using zuul? 19:19:54 fungi: monty asked for name ideas and I threw out a few and said I liked bridge the most 19:20:00 fungi: more star trek bridge 19:20:10 yah. it's "the bridge" 19:20:11 aha 19:20:11 but it fits that way too :) 19:20:32 patrick stewart is apparently going to reprise jean-luc in a new show. this is relevant information for this meeting. 19:20:34 I could use more reviews on that stack - I'm a little bit stalled at the moment needing to get the install_modules change landed 19:20:38 * fungi musters to his station 19:20:39 should have changed the motd to "make it so" 19:20:41 the big thing to be aware of at the moment is we need to double account hiera data changes on puppetmaster.o.o and brdige.o.o until bridge takes over the puppet triggering 19:20:47 and yes, that 19:21:08 I don't think we're too far away from being able to move triggering to bridge and decommission puppetmaster 19:21:12 corvus: cautiously optomistic about that :) 19:21:16 also, you *are* allowed to call it "bilge" when things go wrong 19:21:28 oh - also, double-accounting any password info in /root too 19:21:49 mordred: maybe we could decide to just use bridge for root/passwords.gpg? 19:21:56 since that's just manual 19:22:06 corvus: that is probably a good idea 19:22:08 yah. I'm fine with that 19:22:17 maybe send a followup to the fyi email thread saying we are doing ^ 19:22:29 mordred: can you do one final sync, then move the old ones out of the way and replace with a README? 19:22:34 maybe let's set u-w on passwords.gpg on puppetmaster just to be sure ... o rthat 19:22:36 yes 19:22:42 ya, something like that :) 19:22:49 * mordred will do that 19:23:24 mordred: I'll also rebuild the rax dns venv on bridge so that ssl works 19:23:38 (I hadn't done that on puppet master yet) 19:24:04 Semi related are the set of changes at topic:puppet-4 19:24:20 I think I'm mostly up to date on my reviews there if someone else is willing tomake a followup pass through 19:24:30 but the vast majority of those changes have been merged at this point 19:24:44 i think we're stuck on ask and ask-staging 19:24:55 since ask-staging seems to not exist right now 19:24:59 oh right, ianw ^ is there a tl;dr for the status on ask-staging? 19:25:11 (I think you had mentioned that there were some things) 19:25:32 oh, hrm ... so i think i was waiting on a change to get numerical hosts for it into puppet 19:25:41 and then probably got distracted 19:26:08 yes, https://review.openstack.org/#/c/558992/ 19:26:46 ianw: ya I left a couple comments on that one last week when I was trying to get the future parser stuff going 19:26:58 ianw: I think we need to set up the hiera groups in puppet for that (and then add hiera data) 19:27:14 cmurphy: would it make sense to skip ask* for now and continue with another service or three? 19:27:35 clarkb: sure 19:27:46 i can do that, we really should look at getting ask.o.o updated and will need the staging host 19:27:57 ianw: ++ 19:28:05 there's an email out there i sent when i investigated issues with ask.o.o 19:28:12 I think we can do the puppet 4 parser work in parallel (on other services) and update ask 19:28:15 i've just been going through site.pp looking for candidates 19:28:40 i have a change to remove stackalytics.o.o since that seesm to not exist anymore https://review.openstack.org/585208 19:28:42 cmurphy: ethercalc, ehterpad, logstash-workers, elasticsearch, paste are favorites of mine fwiw. They tend to be straightforward 19:28:56 clarkb: ether* scare me because nodejs 19:29:24 storyboard-dev? 19:29:42 also there are a few in-flight beaker tests for ether* that i think are still unmerged 19:30:20 cmurphy: we have etherpad-dev too but getting tests happy first is probably a good idea 19:30:20 fungi: storyboard-dev is good except it's slightly less dev than other -dev sites since some people are using it to trial storyboard 19:30:35 also i noticed that odsreg.o.o seems very sad 19:30:50 i thought we removed that 19:31:06 I want to say we did 19:31:12 i need to follow up with ttx to find out if i missed deleting that after he said i could or something 19:31:29 it's still in site.pp 19:31:42 if it's not needed i'll propose a change to remove it 19:31:45 and dns 19:31:50 we should clean that up if possible 19:31:56 i'll take care of it 19:31:59 thanks 19:32:07 after i check that it's really gone from the tenant too 19:33:03 mordred: cmurphy: anything else we should be aware of with the changes that are happening? 19:33:30 nothing from me i think 19:34:10 yeah, not finding odsreg in our tenant, so it's just puppet/dns i need to clean up i guess 19:34:35 #topic General Topics 19:34:44 corvus: Swift logs is back on the agenda 19:35:36 we're very close to being able to switch 19:35:54 we need to land this change: 19:36:03 #link https://review.openstack.org/588383 Add severity info to logstash and filter out DEBUG lines 19:36:13 to the log processor 19:36:24 that should be compatible with the current system too, so is safe to land now 19:36:27 I've reviewed it and has my +2. It should be forward and backward compatible 19:36:29 yup 19:36:58 then i can push up some test changes (to, say, devstack) so we can see how everything will look 19:37:05 then we should be able to switch 19:37:21 so, basically, now's the time to start talking about whether we want to, any concerns folks have, etc. 19:37:34 corvus: we should clarify to openstack release team that this won't affect the releaes artifacts (all of that goes to tarballs still aiui) 19:37:50 I'm surethey will be concered about it impacting the release (but its logs only shold be quite safe) 19:38:05 corvus: is the plan to use vexxhost to start? 19:38:45 currently, it's only easy to use one provider. we can use vexxhost or ovh. i think we should be able to randomly pick one of the two in the near future if we want, but i haven't implemented that yet. 19:39:10 we *might* be able to extend it to support rax's not-swift. no promises. 19:39:35 those are the only things vaguely like swift that i believe we have access to at the moment. 19:40:35 ++ 19:40:57 there are definite downsides to this. severity filtering in the ui will be gone. we'll be storing plain text *and* htmlified logs. and tristanC apparently wanted to use os-loganalyze for some of the machine learning log stuff. 19:41:19 we are trading complexity for reliability and retention 19:41:33 I'm personally happy to use grep myself if I have to in return for having logs for longer period of time 19:42:02 that's how i look at it. and i'd like to look into implementing some of that complexity in javascript if we can. 19:42:31 vim users, you can open http(s) urls directly in vim too. Quite useful for this stuff 19:42:36 I'm sure emacs does similar 19:42:44 yeah, i expect some css/js magic would make it possible to filter in-browser (though the browser would still have to retrieve the full log first) 19:43:32 if we can't, maybe we should revisit the idea of a proxy. tobiash is running one anyway (for access control purposes), so we might want to revisit that. but at the moment at least, i feel like i'd like to try to avoid that. it's where we hit the wall last time we tried swift. 19:43:48 and the implementation gives us a durable url to the index for any given build? 19:44:17 fungi: yeah. https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/logs_78/587178/1/check/tox-py35-on-zuul/1f46ccf/ is the latest test 19:44:21 fungi: until the expiration happens and it is deleted. Couple that with zuul dashboard and you should be able to find builds reasonably well 19:44:51 oh, yeah, icons are still to-do. 19:44:58 I think it's a reasonable tradeoff 19:45:00 meh, icons ;) 19:45:10 especially given the cost to th eproject overall whenver logs.o.o goes south 19:45:18 mordred: ya 19:45:54 we have 15 minutes left, ok if we move on? Please go review the log processor change if you have some cycles 19:46:00 and I agree, we can always run a proxy in the future - OR we could just run a web service that does os-loganalyze/machine-learning like things that is not what we normally route people to for log viewing 19:46:06 yep. 19:46:17 sounds like folks want to move forward still. 19:46:20 clarkb: I already +A'd the processor change 19:46:24 mordred: woot 19:46:27 let me know if you have concerns. or, you know, propose patches to fix them. :) 19:46:55 do we want a redirector of some sort (rather than a proxy) in case we need to move objects to another container or provider or something? or just plan to do phased transitions and consider the loss of that data an acceptable risk in such situations? 19:47:19 fungi: personally, i'd suggest never moving data 19:47:27 if we lose a provider holding logs, we lose those logs 19:48:08 not all that different from the 30 day expiry on mega fs today 19:48:12 since we don't control those domains though, we have to assume domain name changes will never occur, and urls paths will never be rejiggered, right? 19:48:26 fungi: yes, though I think that is sort of the point of those services. 19:48:33 wfm 19:48:35 the nice thing about not having a proxy (at least, in the current system) is that we no longer have a single hostname for logs. each log has its own url. a job that runs today may be stored in vexxhost. tomorrow in ovh. 19:48:38 fungi: I would be surprised if they change those ever 19:48:55 so us changing log-storageprovider 19:48:57 gah 19:49:02 so us changing log-storage-providers is a non-event 19:49:05 yep, i'm fine with all that as long as we set expectations around it 19:49:40 still no guarantee of any particular retention in cases where we have to move off of a provider we were using 19:49:48 yup 19:49:54 (thinking ahead to a broadly distributed object storage future for logging) 19:50:13 mmm. broadly distributed 19:50:22 fungi: yeah, we may want to not enable log storage on a provider until we have a reasonable expectation they'll stick around for 6 months :) 19:50:29 heh 19:50:32 good call! 19:50:48 Really quickly, the winterscale naming thread hsa an update from jbryce with a proposal. Would appreciate all yall's feedback. I think we are also keeping it to that private thread we are on for now while some last details are sorted. 19:51:21 feedback that has been provided so far is positive though. I am excited :) 19:51:26 ++ 19:51:34 yeah, hopefully we can make that public again soon, but it's good to know there's progress 19:51:39 * mordred is excited - and also excited to get to the point where we can talk about it publically 19:51:44 ++ 19:52:03 #topic Open Discussion 19:52:57 Couple items from me that weren't quite proper agenda worthy: I am taking tomorrow off to go fishing and spend time with family. I expect that I will be properly afk. The other thing is PTG planning is quickly bubbling up to the top of my todo list 19:53:16 I think the config management stuff is getting to the point where we should hav a good idea of what we can dig into in denver 19:53:19 which is good 19:55:02 https://zuul-ci.org/users.html and http://superuser.openstack.org/articles/zuul-case-study-bmw/ are cool 19:55:40 :) 19:55:49 as is https://www.podcastinit.com/zuul-with-monty-taylor-episode-172/ 19:55:51 well, except for that none of the links go to zuul-ci.org. i'll email them i guess :) 19:56:11 corvus: dhellmann and I have been asked to start working on similar from an openstack perspective 19:56:38 happy to inclue others if they are interested (maybe to review?) 19:57:05 cool! 19:57:57 eek, yeah i see one nodepool link gong to our openstack deployment operational docs, and another going to a github mirror 19:58:26 Alright we are basically at time and I have a leftover burger patty to turn into lunch 19:58:33 Thank you everyone 19:58:53 thanks clarkb! 19:58:58 #endmeeting