19:01:20 #startmeeting infra 19:01:21 Meeting started Tue Nov 12 19:01:20 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:22 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:24 The meeting name has been set to 'infra' 19:01:30 #link http://lists.openstack.org/pipermail/openstack-infra/2019-November/006509.html Our Agenda 19:01:37 #topic Announcements 19:02:10 Last week was the summit. I am feeling extra zombie like after that trip. The jet lag has been harsh. I expect others are in a similar position 19:02:33 * fungi mutters, "braiiiiins..." 19:03:20 some folks are still vacationing/travelling too 19:03:59 #topic Actions from last meeting 19:04:08 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-29-19.01.txt minutes from last meeting 19:04:27 The last meeting was a long time ago, no actions recorded 19:05:01 #topic Priority Efforts 19:05:09 #topic OpenDev 19:05:16 We need to renew the opendev.org cert 19:05:36 I think we can manage to do that via LE now 19:05:52 we'll just need a handler that restarts gitea backends when the cert updates (and maybe we can serialize those?) 19:06:11 figuring that out is likely to be my tomorrow task 19:06:31 and we have plenty of time to sort it out and if we can't for whatever reason can always buy another cert for a yer 19:06:38 * AJaeger waves late 19:07:06 yeah, it's still something like 2.5 weeks out 19:07:11 i guess each gitea host will have it's own cert? 19:07:23 ianw: yes I think so 19:08:45 as long as le doesn't scoff at 8 hosts requesting ssl certs for the same name 19:09:07 that should be well within the rate limits iirc. 19:09:46 fungi: i don't think it will matter; each will get a different txt value to put into _acme-challenge, and it should "just work" 19:09:50 famous last words :) 19:10:02 that'll presumably be easier than refactoring things to have ssl terminated on the lb 19:10:36 we can also have an altname that matches each backend specifically to help distinguish them? 19:10:44 true 19:10:49 that may also make LE happier if the records are different each time 19:11:09 if it works the way we've done it for other services, i think that happens by default 19:14:17 #topic Update Config Management 19:14:55 I'm not sure this has been called out elsewhere yet but was mentioned at the PTG. Apparently mordred thinks we shouldn't wholesale move to podman yet 19:15:24 oh, why? 19:15:31 we'll be getting dev builds via the ubuntu ppa for podman and some commands don't work as before. Idea is to try it a bit first with review(-dev) 19:15:57 so, "compatibility" ? 19:16:55 I think we are happy with compatibility since rhel8 docker command is podman so it must be fairly compatible. More just that we aren't sure if the dev packaging from the ppa poses any concerns yet and users will need to get back to using docker commands instead of docker-compose commands and so on 19:17:13 Then when we are happy with the results we can do broader changes 19:17:35 I see... 19:17:48 lack of an interleaved log viewer was noted as an unfortunate loss for switching away from docker-compose 19:19:00 Anything else on this topic or should we move on? 19:19:11 * fungi has nothing 19:19:21 so for new work, like testing out the nodepool-builder images, should we use docker or podman? 19:19:28 or in that case, does it not matter so much? 19:19:42 ianw: aiui mordred thinks we should use docker for now and keep podman to review(-dev) 19:20:04 but I may have misunderstood and maybe he meant just don't go changing existing docker usage yet 19:20:14 my hunch is that its probably ok for any new use of containers 19:21:12 ok 19:22:47 #topic Storyboard 19:23:10 fungi: ^ anything new from storyboard? I expect not given the summit 19:23:45 there was an sb onboarding session at the ptg, but nobody showed up so diablo_rojo_phon and tonyb turned it into an openstack technical election officials working session 19:23:55 other than that, nothing new i'm aware of 19:24:18 #topic General topics 19:24:28 #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 PTG Notes 19:24:41 I/we tried to take notes at the summit/forum/ptg 19:25:09 The item on my todo list before opendev.org LE certification is to write up a summary and send it to the infra list 19:25:51 I did want to point out that the etherpad is there for people to browse too and that if you have questions about specific topics feel free to ping me nowish and I'll add it to the summary or you can followup on that thread when it exists 19:26:43 Our meeting with the gitea maintainers went well. We were able to show them some of our build and test tooling to test out the elasticsearch indexing 19:27:01 from that we discovered that the china golang and google golang proxies had different content for the CORS package :/ 19:27:15 that's scary 19:27:44 they also acknowledged that the performance issues we've hit are a major problem and they have other users struggling with that too. I hope that means we continue to see improvements with performance 19:28:59 though we also upgraded 19:29:20 and the new version should have at least some performance improvement over what we were running before last week 19:30:54 Also their community is largely on discord and is activ ethere if we want to communicate with them further 19:31:04 I've got a discord account somewhere, but I should prbably set that up for gitea now 19:32:22 and I'll try to write all this up properly between now and tomorrow sometime 19:32:47 fungi: seems like there were wiki changes prior to leaving for the summit. Were you able to check on those? 19:32:56 nope 19:33:21 i mean yep some changes merged, nope i did not find time yet to follow up further 19:34:20 For static.openstack.org mnaser volunteered to do the job changes on the openstack side of things 19:35:10 ianw: AJaeger ^ I think that means our next step is to create afs volumes, then mnaser can upate jobs to publish to afs and static.o.o, we verify content, then flip things over site by site? 19:35:27 clarkb: I did the first jobs already, happy to hear that mnaser will do the rest! 19:35:43 clarkb: https://review.opendev.org/#/q/status:open+project:openstack/project-config+branch:master+topic:static-services 19:35:47 thanks! 19:35:49 ok, i can look at the volumes for ^ 19:36:29 getting those jobs done helped to answer a couple of questions and finish the spec ;) 19:36:43 I think we should be able to have both sites exist in parallel and then flip dns when we are happy with the new side of things 19:36:45 please check also whether those jobs are really doing what we wanted 19:38:10 And finally we get to ask.openstack.org 19:38:21 #link http://paste.openstack.org/show/785843/ Some logs 19:38:37 these logs (that frickler collected?) seem to show there are segfaults 19:40:30 "or similar nasty error" 19:40:35 that's oddly unspecific 19:41:09 one thing we can try if it is the log rotate that causes this is to ensure we are using copy truncate 19:41:13 and not the rotate in place behavior 19:42:04 systemd-sysv-generator[6394]: Overwriting existing symlink /run/systemd/generator.late/jetty8.service with real service. 19:42:08 what does that mean? 19:42:22 it must be coming from puppet, it's very frequent 19:43:11 ianw: if jetty is a sysv init script that may be a result of it setting up the compat shims 19:45:15 there's a /etc/init.d/jetty8~ file 19:45:51 i've removed that 19:45:55 k 19:46:06 /etc/init.d/jetty is a symlink to /etc/init.d/jetty8 19:46:53 istr doing that as a total hack on the xenial upgrade 19:47:20 https://cgit.freedesktop.org/systemd/systemd/tree/src/sysv-generator/sysv-generator.c?id=4e5589836c9e143796c3f3d81e67ab7a9209e2b0#n166 suggests it's the backup file though 19:49:29 I've not yet had time to look myself other than reading that paste of logs quickly 19:49:42 I'll add it to the todo list to see if anything obvious jumps out at me though 19:50:24 #topic Open Discussion 19:50:28 Anything else? 19:50:33 i'll try and keep an eye too ... but this server really does hang on by a thread :/ 19:50:37 y 19:50:39 er yeah 19:51:22 i've got nothing else 19:51:26 As mentioned before the jet lag has been harsh. I'm trying to get back to a normal schedule but haven't quite managed it yet so will be in and out while I nap :) 19:54:09 #endmeeting