19:01:45 <clarkb> #startmeeting infra
19:01:46 <openstack> Meeting started Tue Feb 26 19:01:45 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:49 <openstack> The meeting name has been set to 'infra'
19:02:18 <gary_perkins> o/ HI!
19:02:46 <ianw> o/
19:03:05 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-February/006289.html
19:03:15 <clarkb> #topic Announcements
19:03:24 <clarkb> OpenStack TC campaigning period is happenign right now
19:03:35 <clarkb> with the voting to start just before Midngith UTC today
19:04:29 <clarkb> *midnight
19:05:07 <clarkb> Keep an eye out for your ballot I guess and you can ping the election officials if you don't get one and expected to
19:05:11 <clarkb> #topic Actions from last meeting
19:05:18 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-19-19.01.txt minutes from last meeting
19:05:50 <clarkb> No actions called out. Thank you to everyone that managed to review the letsencrypt spec
19:06:33 <clarkb> #topic Specs approval
19:06:42 <clarkb> That spec just happens to be the next item of discussion
19:06:50 <clarkb> #link https://review.openstack.org/#/c/587283/ LetsEncrypt Spec
19:07:00 <clarkb> I think this is ready to go up for approval
19:07:23 <clarkb> anyone think we should take more time to refine the spec?
19:08:05 <fungi> we can refine in the implementation from this point on, i expect
19:09:07 <ianw> i've convinced myself at least with the prototype it should work
19:09:16 <ianw> (insert famous last words :)
19:10:04 <clarkb> ok I don't hear any objects. Please review and vote on the spec Before EOD Thurdsay (clarkb local time)
19:10:21 <clarkb> I'll plan to approve the spec if there are no major disagreements between now and then
19:10:30 <clarkb> my typing is not great today
19:11:21 <clarkb> #topic Priority Efforts
19:11:30 <clarkb> #topic Storyboard
19:11:48 <clarkb> looks like placement is going to be headed into storyboard
19:12:05 <clarkb> storyborad also has a lot of interest from outreachy candidates
19:12:10 <fungi> a LOT
19:12:31 <fungi> oh, and we're close to landing the api side of SotK's attachments work
19:12:31 <clarkb> I've been trying to keep up and respond to them when things come up in my timezone
19:12:51 <clarkb> there are current questions about running the unittests locally so if anyone has done thatrecently maybe you can pop over to the storyborad channel and help ivy out
19:12:55 <cmurphy> outreachy applications can be really overwhelming @.@
19:13:13 <fungi> i've asked mordred to weigh in on the swiftclient usage there. not sure whether openstacksdk would make it harder to use storyboard with stand-alone swiftclient
19:13:20 <fungi> er, i mean stand-alone swift
19:13:29 <clarkb> And then maybe we can do a followon to the docs because running the script to set things up doesn't seem to have set things up (or maybe its local issue/ I don't know)
19:13:31 <mordred> it shouldn't
19:13:45 <mordred> and if it does it's likely something we can/should fix in sdk
19:14:02 <fungi> #link https://review.openstack.org/633365 Add a Swift storage backend implementation
19:14:08 <fungi> for anybody who wants to take a look
19:14:17 <mordred> sdk also transparently handles large object chunking and multi-threaded uploading for the user
19:14:32 <fungi> cmurphy: more overwhelming than we've seen in previous years. like dozens of applicants coming out of the woodwork this time
19:14:59 <fungi> mordred: well, this implementation doesn't upload for you, it hands off an upload url
19:15:01 <mordred> but it looks like, from initial looking, that the approach there is to get an upload url from swift that the api consumer should PUT to, I'm guessing
19:15:04 <mordred> yeah
19:15:28 <fungi> basically just want sb to coordinate and track attachments, not funnel the attachments themselves through sb
19:15:52 <mordred> that shoudl probably use the tempurl middleware support - the end user is unlikley to have auth creds for the swift
19:15:58 <mordred> I'll leave some comments
19:16:39 <fungi> the corresponding storyboard-webclient changes are up too
19:17:43 <clarkb> mordred: thank you for helping with that
19:17:51 <clarkb> Anything else storyboard related?
19:17:59 <fungi> #link https://review.openstack.org/633611 Add a resource subcontroller for attachments
19:18:05 <diablo_rojo_> None from me
19:18:08 <fungi> the webclient side of the implementation starts there
19:18:42 <fungi> yeah, i don't have anything else exciting on the sb front for now
19:18:53 <clarkb> #topic Update Config Management
19:19:15 <clarkb> I've merged the last futureparser change (ask.o.o) which ran into a problem with the postgres puppet module
19:19:26 <clarkb> ianw's old chagne to bump that module to v4.8.0 shoud fix this
19:19:39 <clarkb> ianw: is that something you want to approve or should I go ahead and do that after our meeting?
19:19:54 <clarkb> #link https://review.openstack.org/#/c/558995/ Update postgres module to fix usage with puppet futureparser
19:20:12 <ianw> oh, it's been a long time, that's a 5* series URL :)
19:20:18 <clarkb> ianw: indeed
19:20:28 <clarkb> once that is sorted I think we are ready to start doing the puppet 4 upgrades too
19:20:41 <ianw> i think that's ok, iirc it was the minimum viable update?
19:20:58 <clarkb> ianw: minimum viable for xenial. 4.3.0 would fix the puppet4 issues according to the chagnelog
19:21:10 <clarkb> I think we should just make it xenial ready since that is a topic for later in the meeting
19:21:38 <clarkb> On the ansible + docker front corvus has been creating a consistent template for deploying services with host networking using docker compose
19:21:38 <ianw> yeah, this was in service of ask.o.o updating, so let's do it i say
19:21:58 <fungi> i concur
19:21:58 <corvus> we have two services configured for that, zero of which are running
19:22:04 <corvus> but should be rsn.  :)
19:22:07 <clarkb> corvus: have you had a chance to look at the service restart when image updates thing I pointed out?
19:22:16 <corvus> oh sorry, 3 services, 2 of which are not running :)
19:22:20 <clarkb> corvus: ya overall I think I like it because its quite consistent so far
19:22:30 <clarkb> you don't have to change much to deploy foo instead of bar
19:22:42 <corvus> clarkb: erm, what was the issue with that again?
19:22:56 <clarkb> corvus: docker-compose up -d will restart any service that has a new image available
19:23:05 <fungi> i could go for deploying to a bar right about now
19:23:10 <clarkb> corvus: and sicne you are using foo-image:latest it will see newer images
19:23:22 <clarkb> I guess one fix is to fix the image version.
19:23:29 <corvus> deploying latest sounds great to me :)
19:23:42 <clarkb> In the case of multiple redundant services (like gitea) we mighte be able to get away with serializing the docker compose up commands
19:23:51 <clarkb> then haproxy will sort things out for the most part
19:24:19 <clarkb> its something we'll want to have an answer for when these services go into production. But we don't need to solve it in this meeting
19:24:28 <corvus> well, i mean, i think we have the answer
19:24:57 <corvus> i wrote a docker-compose file that says "run the latest image" and docker-compose is making sure that the latest image is running, so w00t.
19:25:33 <clarkb> right, but those restarts will be user visible if they coincide with a request right?
19:25:43 <corvus> yes, but for non-ha services, that's life
19:25:54 <corvus> i do agree that for the ha services, we should tell ansible to run serially
19:26:05 <corvus> that's more of an ansible thing than a docker thing though
19:26:12 <clarkb> ya
19:26:36 <corvus> i do think that our gitea servers should always run the latest gitea image we built
19:26:49 <corvus> i think this gets to the fact that we need to rework our system-config playbooks
19:26:49 <mordred> yeah. as long as the ansible is serial I think that shoudl work out
19:26:55 <mordred> corvus: ++
19:26:56 <corvus> we need to stop hanging everything off of base
19:27:09 <fungi> for ha services, running serially *with a delay* might even be called for, just because there will be some lag in lb checks
19:27:12 <corvus> we need the gerrit/zuul/git/gitea playbook.
19:27:33 <corvus> fungi: we can also execute haproxy commands
19:27:41 <corvus> this is the real advantage of using ansible
19:28:01 <clarkb> ya should be able to coordinate the entries in haproxy so that it all happens in a safe manner
19:28:04 <corvus> take a member offline, upgrade it, put it back online
19:28:12 <clarkb> looks like https://review.openstack.org/#/c/604925/ is in merge conflict again I'll rebase it again
19:28:22 <clarkb> (thats the bit needed to have zuul trigger playbooks on bridge)
19:28:45 <corvus> related to this, just to keep folks up to date --
19:28:45 <fungi> corvus: ooh, i do like the idea of orchestrating pool management in haproxy during updates
19:28:56 <corvus> we do not yet have speculative docker images
19:29:08 <mordred> sad trombone
19:29:24 <fungi> that's waiting for zuul to support provider affinity for dependent jobs, right?
19:29:24 <corvus> we managed to demonstrate that working last week, but we've run into a problem which requires a zuul/nodepool feature before we can use it in prod
19:29:46 <corvus> so keep in mind we're still in the world where you need to land changes to images first before you can use them
19:29:54 <corvus> (aka the real world everyone else lives in)
19:30:19 <corvus> boo ^
19:30:26 <fungi> soon we can reject their reality and substitute our own
19:30:32 <corvus> yay ^
19:30:34 <clarkb> if only the internet was ipv6 ready
19:30:40 <fungi> and then invite them over for brunch
19:30:57 <corvus> and yes, the fact that we have non ipv6 providers is the proximate cause.  :(
19:31:10 <corvus> let's not get started
19:31:20 <clarkb> alright anything else related to config management before we talk opendev?
19:31:25 <corvus> nak
19:31:35 <clarkb> #topic OpenDev
19:31:38 <fungi> (which will also be somewhat about config management, i guess)
19:32:02 <corvus> as promised (2 weeks ago) i refreshed the tasks needed to make opendev-gerrit happen
19:32:10 <mordred> \o/
19:32:12 <corvus> #link opendev-gerrit tasks https://storyboard.openstack.org/#!/story/2004627
19:32:25 <clarkb> The openstack berlin ops meetup planning etherpad (or one of them), https://etherpad.openstack.org/p/BER19-OPS-DUDES, has a section to talk about opendev and opinionate on it
19:32:39 <clarkb> I don't plan to be in berlin but if anyone else is there that might be a good thing to sit in on
19:32:45 <fungi> now we can all start signing up for the opendev-gerrit tasks
19:32:53 <corvus> so what would be really great, is if everyone would sign up for those
19:32:58 <clarkb> corvus: thank you for getting that posted
19:33:09 <corvus> i will not be doing all of those myself
19:33:31 <clarkb> I'll likely jump on the git:// task
19:33:32 <corvus> so if no one signs up for those, it won't be done.
19:33:39 <fungi> i will try to at least grab some of the easier ones
19:33:41 <clarkb> as a devstack core (how did that happen) I'm probably well positioned for that one
19:33:55 <fungi> but seems like clarkb's already got the easiest one ;)
19:33:56 <corvus> please use the storyboard assignment feature to put your name on it
19:34:08 <clarkb> fungi: I wish that was an easy one. You can have it :P
19:34:15 <fungi> heh
19:34:22 <corvus> most of them are easy
19:34:36 <corvus> the gitea servers are just launching nodes at this point
19:34:45 <corvus> and i already wrote the .htaccess file to do all the redirects
19:35:25 <clarkb> #action Everyone please sign up for tasks at https://storyboard.openstack.org/#!/story/2004627 to help with the opendev git hosting and gitea work
19:35:40 <fungi> thanks corvus! i'm really excited about this
19:35:42 <corvus> erm
19:35:58 <ianw> is that page working?
19:36:00 <corvus> oh, weird.  that link took a really long time to load for me
19:36:11 <ianw> oh, yeah, maybe i'm just impatient
19:37:09 <clarkb> Anything else to add on the opendev topic?
19:37:21 <fungi> we do need someone with rdbms smarts to help us optimize our queries for sb ;)
19:37:40 <corvus> clarkb: nak
19:37:56 <fungi> (would be a great opportunity to works one-on-one with an outreacy intern too!)
19:38:08 <fungi> wow, i can't type today
19:38:27 <clarkb> #topic General Topics
19:38:30 <clarkb> fungi: neither can I
19:39:01 <clarkb> First up I'd like to bring up the Trusty Upgrades situation again. Last week I realized I didn't really formulate enough thought or discussion around whether or not we thought a sprint would be helpful
19:39:17 <clarkb> I have been picking off a service at a time and trying to get it finished up. Currently doing health.openstack.org
19:39:31 <clarkb> We have about 2 more months of support on Trusty.
19:39:44 <fungi> there's not a lot left to tackle for a sprint, is there?
19:39:45 <clarkb> It would be really helpful if others could grab things off of https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup and help to upgrade them
19:40:09 <mordred> clarkb: all the fun ones
19:40:10 <clarkb> fungi: afs, groups, lists, ask, static, the wiki, and graphite and status
19:40:17 <clarkb> ya the list of easy things is shrinking :)
19:40:21 <fungi> lists.o.o is going to need another in-place ubuntu upgrade, i expect, so we don't have to reset its ip address's reputation
19:40:31 <clarkb> fungi: and corvus suggested we do that same for afs yesterday
19:40:34 <fungi> sounds like ask.o.o may also be an in-place
19:40:52 <fungi> oh, that was afs maybe not ask
19:41:23 <mordred> fungi: they're both three leter words starting with a
19:41:23 <mordred> so - you know, close enough
19:41:30 <clarkb> So my big ask is please try to find some time for this. And does anyone think a sprint would be helpful?
19:41:48 <clarkb> openstack feature freeze is next week. I don't know when RCs happen but we could do a sprint the week after RCs potentially
19:42:02 <clarkb> I'm happy to organize a sprint if we think that would be useful
19:42:05 <fungi> it's possible groups.o.o can be decommissioned. i'll make inquiries
19:42:38 <mordred> that would be less work than upgrading
19:42:48 <fungi> wiki is another one which will probably be a snapshot and in-place upgrade unless someone has time to finish the puppeting
19:43:10 <clarkb> fungi: we should probably start annotating that etherpad with this info
19:43:39 <fungi> static, graphite and status are probably not hard. just involve downtime for some stuff and moving cinder devices or rsync'ing data
19:43:47 <clarkb> fungi: yup
19:43:48 <mordred> fungi: I think snapshot/in-place sounds better - because at this point I think converting that to docker is likely to be more pleasant than finishing the puppet
19:43:49 <fungi> and yes, i think that's a great idea
19:43:51 <mordred> (for wiki)
19:44:05 <fungi> mordred: or converting it to gitea
19:44:15 <mordred> fungi: or that.
19:44:39 <clarkb> with the sprint idea any rough show of hands for people that would be able to dedicate say 3 days to that? or at least a good portion of a 3 day chunk of time?
19:45:01 <ianw> o/ i could, modulo when i'm awake :)
19:45:23 <corvus> maybe 2 days? :)
19:46:21 <clarkb> RC1 is targeted for march 22nd ish
19:47:08 <clarkb> which would make march 25-29 the rough time I have in mind. I can ping the openstack release team and ask them if then or ealier is better then suggest a couple options on the infra list
19:47:09 <mordred> I'm travelling to europe for 2 conferences march 26-april 4 - but could do virtual sprint either around then or as best I could given timezones
19:47:25 <clarkb> its possible they would want us to do the things before RCs
19:47:34 <clarkb> ok sounds liek the number is greater than just me :)
19:47:50 <clarkb> I'll talk to the release team and suggest options on the infra list in the near future. Thanks
19:47:55 <fungi> if folks are in favor of a sprint i can pitch in but am spread a bit thin to dedicate 3 days doing primarily that
19:48:05 <clarkb> The other item related to this is the puppetmaster.
19:48:18 <fungi> though i'm happy to continue to work on upgrading servers outside of or without a sprint too
19:48:19 <clarkb> Last call if you need any scripts or data or anything off of that server
19:48:34 <clarkb> otherwise I'd like to delete it this week (I'm happy to do the deleting just want to make sure people know this is happening()
19:48:42 <fungi> i need nothing from the old puppetmaster, but snapshot it first?
19:48:48 <clarkb> I can snapshot it first
19:48:53 <fungi> thanks!
19:48:58 <clarkb> #info clarkb snapshot puppetmaster before deleting the old server
19:49:41 <clarkb> Last item on the list is thoughts on Infra/OpenDev onboarding and/or update sessions at the denver summit
19:49:59 <clarkb> I'm leaning towards no on the onboarding session (I think the return on those has been low and spots are limited)
19:50:17 <clarkb> but askign for an update session to talk about all the big changes around config management and docker and ansible and gitea and even k8s
19:50:28 <corvus> clarkb: i support that plan
19:50:37 <mordred> ++
19:51:56 <clarkb> Alright that is all we had on the agenda.
19:51:59 <clarkb> #topic Open Discussion
19:52:01 <clarkb> Anything else?
19:52:18 <fungi> sounds good to me, though we could also consider abusing the onboarding session to do more of an inservice for folks who want to come learn how to use what we've built... though that might be better saved for once we're further along with the opendev transition
19:52:43 <corvus> which is only going to happen if people sign up for tasks on https://storyboard.openstack.org/#!/story/2004627
19:52:49 <corvus> i'm still the only name on there
19:52:59 <clarkb> corvus: its on my list for after the meeting :)
19:55:03 <clarkb> I'm not hearing anything else so I'll end the meeting now and we can all go sign up on corvus' list. Thanks everyone
19:55:09 <clarkb> #endmeeting