19:01:45 #startmeeting infra 19:01:46 Meeting started Tue Feb 26 19:01:45 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:47 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:49 The meeting name has been set to 'infra' 19:02:18 o/ HI! 19:02:46 o/ 19:03:05 #link http://lists.openstack.org/pipermail/openstack-infra/2019-February/006289.html 19:03:15 #topic Announcements 19:03:24 OpenStack TC campaigning period is happenign right now 19:03:35 with the voting to start just before Midngith UTC today 19:04:29 *midnight 19:05:07 Keep an eye out for your ballot I guess and you can ping the election officials if you don't get one and expected to 19:05:11 #topic Actions from last meeting 19:05:18 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-19-19.01.txt minutes from last meeting 19:05:50 No actions called out. Thank you to everyone that managed to review the letsencrypt spec 19:06:33 #topic Specs approval 19:06:42 That spec just happens to be the next item of discussion 19:06:50 #link https://review.openstack.org/#/c/587283/ LetsEncrypt Spec 19:07:00 I think this is ready to go up for approval 19:07:23 anyone think we should take more time to refine the spec? 19:08:05 we can refine in the implementation from this point on, i expect 19:09:07 i've convinced myself at least with the prototype it should work 19:09:16 (insert famous last words :) 19:10:04 ok I don't hear any objects. Please review and vote on the spec Before EOD Thurdsay (clarkb local time) 19:10:21 I'll plan to approve the spec if there are no major disagreements between now and then 19:10:30 my typing is not great today 19:11:21 #topic Priority Efforts 19:11:30 #topic Storyboard 19:11:48 looks like placement is going to be headed into storyboard 19:12:05 storyborad also has a lot of interest from outreachy candidates 19:12:10 a LOT 19:12:31 oh, and we're close to landing the api side of SotK's attachments work 19:12:31 I've been trying to keep up and respond to them when things come up in my timezone 19:12:51 there are current questions about running the unittests locally so if anyone has done thatrecently maybe you can pop over to the storyborad channel and help ivy out 19:12:55 outreachy applications can be really overwhelming @.@ 19:13:13 i've asked mordred to weigh in on the swiftclient usage there. not sure whether openstacksdk would make it harder to use storyboard with stand-alone swiftclient 19:13:20 er, i mean stand-alone swift 19:13:29 And then maybe we can do a followon to the docs because running the script to set things up doesn't seem to have set things up (or maybe its local issue/ I don't know) 19:13:31 it shouldn't 19:13:45 and if it does it's likely something we can/should fix in sdk 19:14:02 #link https://review.openstack.org/633365 Add a Swift storage backend implementation 19:14:08 for anybody who wants to take a look 19:14:17 sdk also transparently handles large object chunking and multi-threaded uploading for the user 19:14:32 cmurphy: more overwhelming than we've seen in previous years. like dozens of applicants coming out of the woodwork this time 19:14:59 mordred: well, this implementation doesn't upload for you, it hands off an upload url 19:15:01 but it looks like, from initial looking, that the approach there is to get an upload url from swift that the api consumer should PUT to, I'm guessing 19:15:04 yeah 19:15:28 basically just want sb to coordinate and track attachments, not funnel the attachments themselves through sb 19:15:52 that shoudl probably use the tempurl middleware support - the end user is unlikley to have auth creds for the swift 19:15:58 I'll leave some comments 19:16:39 the corresponding storyboard-webclient changes are up too 19:17:43 mordred: thank you for helping with that 19:17:51 Anything else storyboard related? 19:17:59 #link https://review.openstack.org/633611 Add a resource subcontroller for attachments 19:18:05 None from me 19:18:08 the webclient side of the implementation starts there 19:18:42 yeah, i don't have anything else exciting on the sb front for now 19:18:53 #topic Update Config Management 19:19:15 I've merged the last futureparser change (ask.o.o) which ran into a problem with the postgres puppet module 19:19:26 ianw's old chagne to bump that module to v4.8.0 shoud fix this 19:19:39 ianw: is that something you want to approve or should I go ahead and do that after our meeting? 19:19:54 #link https://review.openstack.org/#/c/558995/ Update postgres module to fix usage with puppet futureparser 19:20:12 oh, it's been a long time, that's a 5* series URL :) 19:20:18 ianw: indeed 19:20:28 once that is sorted I think we are ready to start doing the puppet 4 upgrades too 19:20:41 i think that's ok, iirc it was the minimum viable update? 19:20:58 ianw: minimum viable for xenial. 4.3.0 would fix the puppet4 issues according to the chagnelog 19:21:10 I think we should just make it xenial ready since that is a topic for later in the meeting 19:21:38 On the ansible + docker front corvus has been creating a consistent template for deploying services with host networking using docker compose 19:21:38 yeah, this was in service of ask.o.o updating, so let's do it i say 19:21:58 i concur 19:21:58 we have two services configured for that, zero of which are running 19:22:04 but should be rsn. :) 19:22:07 corvus: have you had a chance to look at the service restart when image updates thing I pointed out? 19:22:16 oh sorry, 3 services, 2 of which are not running :) 19:22:20 corvus: ya overall I think I like it because its quite consistent so far 19:22:30 you don't have to change much to deploy foo instead of bar 19:22:42 clarkb: erm, what was the issue with that again? 19:22:56 corvus: docker-compose up -d will restart any service that has a new image available 19:23:05 i could go for deploying to a bar right about now 19:23:10 corvus: and sicne you are using foo-image:latest it will see newer images 19:23:22 I guess one fix is to fix the image version. 19:23:29 deploying latest sounds great to me :) 19:23:42 In the case of multiple redundant services (like gitea) we mighte be able to get away with serializing the docker compose up commands 19:23:51 then haproxy will sort things out for the most part 19:24:19 its something we'll want to have an answer for when these services go into production. But we don't need to solve it in this meeting 19:24:28 well, i mean, i think we have the answer 19:24:57 i wrote a docker-compose file that says "run the latest image" and docker-compose is making sure that the latest image is running, so w00t. 19:25:33 right, but those restarts will be user visible if they coincide with a request right? 19:25:43 yes, but for non-ha services, that's life 19:25:54 i do agree that for the ha services, we should tell ansible to run serially 19:26:05 that's more of an ansible thing than a docker thing though 19:26:12 ya 19:26:36 i do think that our gitea servers should always run the latest gitea image we built 19:26:49 i think this gets to the fact that we need to rework our system-config playbooks 19:26:49 yeah. as long as the ansible is serial I think that shoudl work out 19:26:55 corvus: ++ 19:26:56 we need to stop hanging everything off of base 19:27:09 for ha services, running serially *with a delay* might even be called for, just because there will be some lag in lb checks 19:27:12 we need the gerrit/zuul/git/gitea playbook. 19:27:33 fungi: we can also execute haproxy commands 19:27:41 this is the real advantage of using ansible 19:28:01 ya should be able to coordinate the entries in haproxy so that it all happens in a safe manner 19:28:04 take a member offline, upgrade it, put it back online 19:28:12 looks like https://review.openstack.org/#/c/604925/ is in merge conflict again I'll rebase it again 19:28:22 (thats the bit needed to have zuul trigger playbooks on bridge) 19:28:45 related to this, just to keep folks up to date -- 19:28:45 corvus: ooh, i do like the idea of orchestrating pool management in haproxy during updates 19:28:56 we do not yet have speculative docker images 19:29:08 sad trombone 19:29:24 that's waiting for zuul to support provider affinity for dependent jobs, right? 19:29:24 we managed to demonstrate that working last week, but we've run into a problem which requires a zuul/nodepool feature before we can use it in prod 19:29:46 so keep in mind we're still in the world where you need to land changes to images first before you can use them 19:29:54 (aka the real world everyone else lives in) 19:30:19 boo ^ 19:30:26 soon we can reject their reality and substitute our own 19:30:32 yay ^ 19:30:34 if only the internet was ipv6 ready 19:30:40 and then invite them over for brunch 19:30:57 and yes, the fact that we have non ipv6 providers is the proximate cause. :( 19:31:10 let's not get started 19:31:20 alright anything else related to config management before we talk opendev? 19:31:25 nak 19:31:35 #topic OpenDev 19:31:38 (which will also be somewhat about config management, i guess) 19:32:02 as promised (2 weeks ago) i refreshed the tasks needed to make opendev-gerrit happen 19:32:10 \o/ 19:32:12 #link opendev-gerrit tasks https://storyboard.openstack.org/#!/story/2004627 19:32:25 The openstack berlin ops meetup planning etherpad (or one of them), https://etherpad.openstack.org/p/BER19-OPS-DUDES, has a section to talk about opendev and opinionate on it 19:32:39 I don't plan to be in berlin but if anyone else is there that might be a good thing to sit in on 19:32:45 now we can all start signing up for the opendev-gerrit tasks 19:32:53 so what would be really great, is if everyone would sign up for those 19:32:58 corvus: thank you for getting that posted 19:33:09 i will not be doing all of those myself 19:33:31 I'll likely jump on the git:// task 19:33:32 so if no one signs up for those, it won't be done. 19:33:39 i will try to at least grab some of the easier ones 19:33:41 as a devstack core (how did that happen) I'm probably well positioned for that one 19:33:55 but seems like clarkb's already got the easiest one ;) 19:33:56 please use the storyboard assignment feature to put your name on it 19:34:08 fungi: I wish that was an easy one. You can have it :P 19:34:15 heh 19:34:22 most of them are easy 19:34:36 the gitea servers are just launching nodes at this point 19:34:45 and i already wrote the .htaccess file to do all the redirects 19:35:25 #action Everyone please sign up for tasks at https://storyboard.openstack.org/#!/story/2004627 to help with the opendev git hosting and gitea work 19:35:40 thanks corvus! i'm really excited about this 19:35:42 erm 19:35:58 is that page working? 19:36:00 oh, weird. that link took a really long time to load for me 19:36:11 oh, yeah, maybe i'm just impatient 19:37:09 Anything else to add on the opendev topic? 19:37:21 we do need someone with rdbms smarts to help us optimize our queries for sb ;) 19:37:40 clarkb: nak 19:37:56 (would be a great opportunity to works one-on-one with an outreacy intern too!) 19:38:08 wow, i can't type today 19:38:27 #topic General Topics 19:38:30 fungi: neither can I 19:39:01 First up I'd like to bring up the Trusty Upgrades situation again. Last week I realized I didn't really formulate enough thought or discussion around whether or not we thought a sprint would be helpful 19:39:17 I have been picking off a service at a time and trying to get it finished up. Currently doing health.openstack.org 19:39:31 We have about 2 more months of support on Trusty. 19:39:44 there's not a lot left to tackle for a sprint, is there? 19:39:45 It would be really helpful if others could grab things off of https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup and help to upgrade them 19:40:09 clarkb: all the fun ones 19:40:10 fungi: afs, groups, lists, ask, static, the wiki, and graphite and status 19:40:17 ya the list of easy things is shrinking :) 19:40:21 lists.o.o is going to need another in-place ubuntu upgrade, i expect, so we don't have to reset its ip address's reputation 19:40:31 fungi: and corvus suggested we do that same for afs yesterday 19:40:34 sounds like ask.o.o may also be an in-place 19:40:52 oh, that was afs maybe not ask 19:41:23 fungi: they're both three leter words starting with a 19:41:23 so - you know, close enough 19:41:30 So my big ask is please try to find some time for this. And does anyone think a sprint would be helpful? 19:41:48 openstack feature freeze is next week. I don't know when RCs happen but we could do a sprint the week after RCs potentially 19:42:02 I'm happy to organize a sprint if we think that would be useful 19:42:05 it's possible groups.o.o can be decommissioned. i'll make inquiries 19:42:38 that would be less work than upgrading 19:42:48 wiki is another one which will probably be a snapshot and in-place upgrade unless someone has time to finish the puppeting 19:43:10 fungi: we should probably start annotating that etherpad with this info 19:43:39 static, graphite and status are probably not hard. just involve downtime for some stuff and moving cinder devices or rsync'ing data 19:43:47 fungi: yup 19:43:48 fungi: I think snapshot/in-place sounds better - because at this point I think converting that to docker is likely to be more pleasant than finishing the puppet 19:43:49 and yes, i think that's a great idea 19:43:51 (for wiki) 19:44:05 mordred: or converting it to gitea 19:44:15 fungi: or that. 19:44:39 with the sprint idea any rough show of hands for people that would be able to dedicate say 3 days to that? or at least a good portion of a 3 day chunk of time? 19:45:01 o/ i could, modulo when i'm awake :) 19:45:23 maybe 2 days? :) 19:46:21 RC1 is targeted for march 22nd ish 19:47:08 which would make march 25-29 the rough time I have in mind. I can ping the openstack release team and ask them if then or ealier is better then suggest a couple options on the infra list 19:47:09 I'm travelling to europe for 2 conferences march 26-april 4 - but could do virtual sprint either around then or as best I could given timezones 19:47:25 its possible they would want us to do the things before RCs 19:47:34 ok sounds liek the number is greater than just me :) 19:47:50 I'll talk to the release team and suggest options on the infra list in the near future. Thanks 19:47:55 if folks are in favor of a sprint i can pitch in but am spread a bit thin to dedicate 3 days doing primarily that 19:48:05 The other item related to this is the puppetmaster. 19:48:18 though i'm happy to continue to work on upgrading servers outside of or without a sprint too 19:48:19 Last call if you need any scripts or data or anything off of that server 19:48:34 otherwise I'd like to delete it this week (I'm happy to do the deleting just want to make sure people know this is happening() 19:48:42 i need nothing from the old puppetmaster, but snapshot it first? 19:48:48 I can snapshot it first 19:48:53 thanks! 19:48:58 #info clarkb snapshot puppetmaster before deleting the old server 19:49:41 Last item on the list is thoughts on Infra/OpenDev onboarding and/or update sessions at the denver summit 19:49:59 I'm leaning towards no on the onboarding session (I think the return on those has been low and spots are limited) 19:50:17 but askign for an update session to talk about all the big changes around config management and docker and ansible and gitea and even k8s 19:50:28 clarkb: i support that plan 19:50:37 ++ 19:51:56 Alright that is all we had on the agenda. 19:51:59 #topic Open Discussion 19:52:01 Anything else? 19:52:18 sounds good to me, though we could also consider abusing the onboarding session to do more of an inservice for folks who want to come learn how to use what we've built... though that might be better saved for once we're further along with the opendev transition 19:52:43 which is only going to happen if people sign up for tasks on https://storyboard.openstack.org/#!/story/2004627 19:52:49 i'm still the only name on there 19:52:59 corvus: its on my list for after the meeting :) 19:55:03 I'm not hearing anything else so I'll end the meeting now and we can all go sign up on corvus' list. Thanks everyone 19:55:09 #endmeeting