19:01:15 #startmeeting infra 19:01:16 Meeting started Tue Dec 18 19:01:15 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:19 The meeting name has been set to 'infra' 19:01:23 o/ 19:01:34 o/ Ho ho ho! :) 19:01:35 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:42 merry :) happy holidays to my US friends :) 19:01:59 it's still hallowe'en here 19:01:59 I wonder if I should start linking to my email thread archive instead of the wiki 19:02:10 clarkb: yes, please! 19:02:10 in any case we have an agenda and I sent it out about 24 hours ago 19:02:22 makes it seem a little more formalized 19:02:43 #link http://lists.openstack.org/pipermail/openstack-infra/2018-December/006255.html 19:02:58 #topic Announcements 19:03:12 As ianw has pointed out we are entering the happy/merry time of year for many of us 19:03:38 The next two regularly scheduled meetings happen on holidays I expect we'll be celebrating so I've cancelled them. Our next meeting will be January 8 19:04:59 If you are taking time off I hope you enjoy it. I'll be traveling myself starting thursday early morning which means I may start winding down work things tomorrow 19:05:15 #topic Actions from last meeting 19:05:27 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-12-11-19.01.txt minutes from last meeting 19:05:48 I've noticed that the actiosn from last meeting section was one I didn't update properly in the email agenda 19:06:05 The two actions we do have from last week are ianw testing github shared account and fungi volunteering for storyboard db moves 19:06:18 I know ianw made progress on github and we've got tiem for that later 19:06:23 fungi: any update on the storyboard db moves? 19:06:47 ahh, no, lemme see where that got to 19:07:50 #link https://review.openstack.org/623290 Run a local MySQL service on StoryBoard servers 19:08:00 seems it's got enough votes but wasn't approved 19:08:14 i'll go ahead and do that now as all it will do is install and start a mysqld 19:08:19 ++, thanks 19:08:34 #topic Specs approval 19:08:51 We've got two specs that I think we can put up for official approval voting 19:08:54 #link https://review.openstack.org/623033 OpenDev Repo Hosting Rework Spec 19:08:59 #link https://review.openstack.org/607377 Storyboard Attachements 19:09:19 i agree those should be ready for council roll call 19:09:22 I'd like to do so now then plan for a longish approval period as I'm unsure of how many days we'll be working over the next couple weeks 19:09:43 normal approval period would be until thursday, right? 19:09:49 is that insufficient? 19:10:11 corvus: it would be. I'm not sure about others but I need to work on packing the family and stuff for a 6am thursday flight :/ 19:10:40 I suppose I could pop my head in once I have internets again after flying and approve changes if others think they will be able to review 19:11:17 perhaps we could ask people to -1 if they need more time? 19:11:22 otherwise proceed as normal? 19:11:24 corvus: thatworks for me 19:12:05 #info Please -1 the above specs if you are not able to review them properly for approval prior to Thursdayish. Clarkb will plan to approve the specs once he has internet again late this week (earliest thursday) 19:12:27 related, i've been working with mordred to get a gitea POC up and running; you can see it here: http://38.108.68.96/ 19:13:02 that's very temporary, but it's something we can play with. i'm happy to talk about the implementation later if folks want. 19:13:09 corvus: thanks 19:13:37 pretend the 'register' button isn't there :) 19:13:49 I'll also call out tristanC and dirks log analysis spec 19:13:51 #link https://review.openstack.org/581214 Anomaly Detection in CI Logs 19:13:59 I don't think this one is ready for approval, but feedback would be good 19:14:22 Linkid also has a wip spec up about hosting a video hosting/streaming service 19:14:36 hi 19:14:37 yes :) 19:14:47 which has some interesting tie-ins to the containerization of services work 19:14:56 https://review.openstack.org/#/c/625450 19:15:07 #link https://review.openstack.org/#/c/625450/ video hosting/streaming spec 19:15:13 ^ the spec about video hosting service 19:15:14 a good one for those who have opinions on what direction we're going in the service container space 19:15:47 cool, i'll review that soon -- i have learned a lot about that lately :) 19:16:19 #topic Priority Efforts 19:16:30 #topic Update Config Management 19:16:56 there has been a bunch of progress and firefighting around ansible and docker and puppet in the last week 19:17:36 Good news is the bootstrapping for docker work is now merged (as of a couple hours ago), we have a script to generate our static inventory too 19:18:04 oh great! which change was that? 19:18:05 Keep in mind the static inventory generation will emit an inventory with our magnum nodes in it, we don't want to use an inventory with those nodes in it, though ansible should fail to ssh into them 19:18:33 #link https://review.openstack.org/#/c/622964/ Inventory generation script 19:18:37 if we apply deltas to config management, we should be able to review that 19:18:48 so we can -1 changes which add magnum nodes :) 19:18:52 ya, mostly a heads up that you'll want to edit the inventory that is generated 19:19:12 but that should be easier than trying to scan all the clouds by hand 19:19:22 ok https://review.openstack.org/605585 is the docker change 19:19:53 I've also started moving the puppet-4 work wioth futureparser enablement forward. Since chances are we'll be running puppet on various services for a while. Next up on the list are the leasticsearch nodes 19:20:02 thank you cmurphy for that stack of chagnes. I'm trying to get through them :) 19:20:24 That did expose that our groups.yaml file is a list of globs not regexes 19:20:51 we should all be careful of that when making changes to the groups.yaml file. + matches a literal + for example and * doesn't apply to the previous character 19:20:53 seems we need to be periodically reminded of that 19:21:06 easy to miss when reviewing 19:21:16 We also discovered a really fun ansible bug 19:21:31 #link https://github.com/ansible/ansible/issues/49969 Ansible bug that crashes current play and returns success 19:22:06 I've worked around ^ in our roles so it no longer affects us, but another warning that nested sequences of include_tasks results in sad ansible that reports happyness 19:22:15 and upstream is working to fix it which is good 19:22:48 Anything else on this effort? 19:22:50 thanks clarkb 19:24:10 #topic Storyboard 19:24:26 As noted above there is the storyboard attachments spec up for approval this week 19:24:50 fungi has the votes necessary to move foward on db migratiosn for storyboard 19:24:51 aside from the spec up for review and me making poor progress on the mysql move, no news i'm aware of 19:25:10 No news besides the spec and the migration of the db really 19:25:15 its been pretty slow going 19:25:16 ok 19:25:28 it is that time of year where everything seems to turn into molasses 19:25:40 Indeed. 19:25:44 #topic OpenDev 19:25:51 #link https://review.openstack.org/#/c/625671/ Publish early opendev web content 19:26:18 Looks like this was just merged? still no website though so we will want to double check on that after hte meeting 19:26:43 In addition to that we now have the ability to host mailing lists at lists.opendev.org 19:27:24 Sounds like corvus has been busy POCing the repo hosting side of things 19:27:38 Looking like early next year might be a sprint to actually hosting things? 19:27:52 very exciting 19:27:58 the poc i linked is running on a 3-node k8s cluster with a shared cephfs 19:28:01 a group of people collaborating around special-purpose virtual machine monitors in rust are our first takers on lists.opendev.org 19:28:25 for anyone wondering how obscure this was going to get right off the bat ;) 19:28:59 mordred and i should be able to push up ansible changes to drive a gitea that way soon 19:29:20 corvus: is that so you can have gerrit replicate as the writer then have gitea mount the filesytem read only in a distributed manner? 19:30:12 clarkb: yes -- but moreover, we would push to gitea in the normal way, so all of its hooks are triggered (so it records and publishes events, etc). but the main thing is that should let us use other features in the future if we want (eg wiki). 19:30:39 can gitea safely do that with multiple instances writing to the same fs? 19:30:41 basically, it would be a real HA gitea system 19:30:44 (things to sort out later I guess) 19:30:51 clarkb: yes 19:30:55 neat 19:31:06 i think it would be our first stateful ha system 19:31:50 the cephfs-in-k8s thing (rook) is pretty neat too. basically we can have all the components HA running inside k8s. it's what k8s was actually designed to do :) 19:32:28 from what i understand of the wiki implementation so far i'm not sure it's actually a fit for some of our current wiki.o.o use cases due to needing explicit editor rights granted from repository owners, but it's possible i'm overlooking what makes it possible to let anyone who authenticates create and edit pages. that said, the ability to drop codesearch.o.o is compelling on its own 19:32:29 ya the one gotcha with ceph run in that manner is that osds are tied to disk so you can't just replace hosts (have to do it in a rolling fashion with intermediate recovery points) 19:33:04 clarkb: yep. but we can handle k8s node failures. 19:33:05 whereas stateless k8s workload you can just move around all you want 19:33:27 fungi: yeah, i'm waving hands on the wiki thing right now. i haven't looked into it at all 19:33:44 fungi: mostly i'm using it as a use case for "let's not back ourselves into a corner so we can't do it later if we want" 19:34:11 oh shoot, i forgot to turn on code searching in the poc 19:34:14 i'll do that after the meeting 19:34:31 alright anything else on OpenDev before we move to the general topics list? 19:35:43 #topic General Topics 19:36:13 ianw volunteered to set up a shared github account that we can use to administer our github things. This has been done and I've confirmed that I can log in to the account 19:36:19 #link https://review.openstack.org/#/c/624531/ ianw's docs update 19:36:33 I think ianw is looking for consensus on ^ before changing the setup in github itself 19:37:05 we've got 2 votes ... i'll give it to say EOD and then i'd like to give the user permissions (and remove my own personal ones) 19:37:52 This simplifies new root onboarding and gets us away from using our personal accounts for this 9which we didn't want to be doing anyway) 19:38:13 sounds good to me; i'll try it out and +2 if no issues 19:39:14 thanks ianw! 19:39:52 Next on the list started as my list of changes I'd like to see merged before I am not around a computer. But they've since merged yesterday 19:39:54 i do wonder if the username is accurate :) 19:40:36 since they have merged I'll go ahead and open it to anyone else to point out changes that would be good to have eyeballs on them before rooters are afk 19:41:27 maybe ansible lint pinning if we end up needing that 19:41:32 though I'm not yet aware of needing that 19:42:19 in any case if you spot important bug fixes please call them out so we can try to get fixes in prior to holidays 19:42:27 And now on to the docker mirroring issues 19:42:48 I'll admit I'm not completely sure I understand what the problem is here. I thought this was a known issue and is why we mirror docker on ports 8081 and 8082 19:43:08 Basically infra has addressed it by having mirrors that handle both cases and jobs need to know which to use based on their version of docker 19:43:27 are there still issues? everyone i saw follow up said switching to the port 8082 mirror solved it for them 19:43:52 frickler: mordred corvus ^ do you know? 19:44:09 it seems the patch has resolved things for us 19:44:14 i don't know anything about dockerhub :( 19:44:31 frickler: have a link to the patch so we can record here if others run into the same issue it may serve as a good template bug fix 19:44:36 and mgoddard also confirm things working for them 19:44:38 s/mirror/proxy/ 19:45:21 #link https://review.openstack.org/625596 Switch to using the pathless :8082 variant of our mirror unconditionally 19:46:04 there is a bit of follow-up because our openstack specific mirror setup shouldn't be known to zuul-jobs 19:46:15 basically newer docker wants things hosted at / and so you can't tell it to talk to host.foo.com:8081/other/root/path 19:46:15 i think that means we can drop 8081 19:46:38 I think there were some projects that might still use 8081 19:46:39 so we have a host.foo.com:8082/ proxy and have kept a host.foo.com:8081/other/path for backward compat 19:46:54 I want to say pabelanger tested older docker against the 8082 one and it didn't work there 19:46:57 frickler: without using that role? 19:47:00 and part of what brought this up is that ubuntu switched to new enough docker to need the new protocol? 19:47:15 corvus: ya tripleo does their own mirror/proxy setup 19:47:37 clarkb: i thought tripleo used the new url too? 19:48:14 corvus: like this http://git.openstack.org/cgit/openstack/kayobe/tree/playbooks/kayobe-overcloud-base/overrides.yml.j2 19:48:21 it's not important i guess; we can check the logs later 19:48:30 corvus: they use the new v2 api on top of the 8081 proxy because they don't actually use docker aiui 19:48:34 corvus: but they don't use that role 19:48:38 its weird 19:48:59 in any case it sounds like we've confirmed that 8082 does fix things and that we should start pushing people towards that address so that we can turn off 8081 19:49:11 probably worth a -discuss email thread for that 19:49:17 fungi: yes, ubuntu backported a new docker into xenial-updates 19:49:23 and we can use apache logs to confirm that people have moved 19:50:04 right, seems like in that case anyone using the port 8081 proxy even from xenial is broken now 19:50:26 so odds are it'll no longer be needed but we can check apache access logs 19:50:35 fungi: ya, the weird one is tripleo beacuse they use 8081 wiht the new v2 url api aiui. And I think they get away with it because their client isn't the docker client 19:50:40 except they install docker from somewhere else than ubuntu repos 19:50:45 so fi we switch tripleo then likely everyone else will be on 8082 19:50:50 k 19:51:22 #topic Open Discussion 19:51:32 Anything else before we get ready to call it a year? 19:53:16 i hope everyone enjoys whatever end-of-year festivities they prefer 19:53:38 i'm celebrating the 8 days of k8s 19:53:45 hah 19:53:54 ubernete 19:54:23 as mentioned earlier starting tomorrow I expect to not be around much. I've got to pack for early thursday flight (and this time with kids which makes logistics a million times more complicated) 19:55:34 i should be around this week. not so much next. 19:55:55 i'm around all this week and intermittently the next 19:56:05 and then around most of the following weel again 19:56:09 week too 19:56:10 fyi i'm pto from EOD today till 07 jan 19:56:20 bold! 19:56:24 i like your style 19:56:46 everyone should enjoy the time they are taking off. I expect that most of openstack and related acitivities will as well 19:57:01 we should remind the release team to not make any releases next week :P 19:57:26 heh 19:57:41 fungi: I still rememebr the year it was pbr or something made a release on the 24th and broke everything for a few days 19:57:56 also the year setuptools and pip released weekend before christmas or was it weekend of new years 19:58:27 oh, yeah, fun times! 19:58:39 Alright sounds like that is it. We'll see you all here on the 8th of January. Until then happy/merry $holiday 19:58:44 #endmeeting