19:01:15 <clarkb> #startmeeting infra
19:01:16 <openstack> Meeting started Tue Dec 18 19:01:15 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:19 <openstack> The meeting name has been set to 'infra'
19:01:23 <diablo_rojo> o/
19:01:34 <gary_perkins> o/ Ho ho ho! :)
19:01:35 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:42 <ianw> merry :)  happy holidays to my US friends :)
19:01:59 <fungi> it's still hallowe'en here
19:01:59 <clarkb> I wonder if I should start linking to my email thread archive instead of the wiki
19:02:10 <fungi> clarkb: yes, please!
19:02:10 <clarkb> in any case we have an agenda and I sent it out about 24 hours ago
19:02:22 <fungi> makes it seem a little more formalized
19:02:43 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2018-December/006255.html
19:02:58 <clarkb> #topic Announcements
19:03:12 <clarkb> As ianw has pointed out we are entering the happy/merry time of year for many of us
19:03:38 <clarkb> The next two regularly scheduled meetings happen on holidays I expect we'll be celebrating so I've cancelled them. Our next meeting will be January 8
19:04:59 <clarkb> If you are taking time off I hope you enjoy it. I'll be traveling myself starting thursday early morning which means I may start winding down work things tomorrow
19:05:15 <clarkb> #topic Actions from last meeting
19:05:27 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-12-11-19.01.txt minutes from last meeting
19:05:48 <clarkb> I've noticed that the actiosn from last meeting section was one I didn't update properly in the email agenda
19:06:05 <clarkb> The two actions we do have from last week are ianw testing github shared account and fungi volunteering for storyboard db moves
19:06:18 <clarkb> I know ianw made progress on github and we've got tiem for that later
19:06:23 <clarkb> fungi: any update on the storyboard db moves?
19:06:47 <fungi> ahh, no, lemme see where that got to
19:07:50 <fungi> #link https://review.openstack.org/623290 Run a local MySQL service on StoryBoard servers
19:08:00 <fungi> seems it's got enough votes but wasn't approved
19:08:14 <fungi> i'll go ahead and do that now as all it will do is install and start a mysqld
19:08:19 <clarkb> ++, thanks
19:08:34 <clarkb> #topic Specs approval
19:08:51 <clarkb> We've got two specs that I think we can put up for official approval voting
19:08:54 <clarkb> #link https://review.openstack.org/623033 OpenDev Repo Hosting Rework Spec
19:08:59 <clarkb> #link https://review.openstack.org/607377 Storyboard Attachements
19:09:19 <fungi> i agree those should be ready for council roll call
19:09:22 <clarkb> I'd like to do so now then plan for a longish approval period as I'm unsure of how many days we'll be working over the next couple weeks
19:09:43 <corvus> normal approval period would be until thursday, right?
19:09:49 <corvus> is that insufficient?
19:10:11 <clarkb> corvus: it would be. I'm not sure about others but I need to work on packing the family and stuff for a 6am thursday flight :/
19:10:40 <clarkb> I suppose I could pop my head in once I have internets again after flying and approve changes if others think they will be able to review
19:11:17 <corvus> perhaps we could ask people to -1 if they need more time?
19:11:22 <corvus> otherwise proceed as normal?
19:11:24 <clarkb> corvus: thatworks for me
19:12:05 <clarkb> #info Please -1 the above specs if you are not able to review them properly for approval prior to Thursdayish. Clarkb will plan to approve the specs once he has internet again late this week (earliest thursday)
19:12:27 <corvus> related, i've been working with mordred to get a gitea POC up and running; you can see it here: http://38.108.68.96/
19:13:02 <corvus> that's very temporary, but it's something we can play with.  i'm happy to talk about the implementation later if folks want.
19:13:09 <clarkb> corvus: thanks
19:13:37 <corvus> pretend the 'register' button isn't there :)
19:13:49 <clarkb> I'll also call out tristanC and dirks log analysis spec
19:13:51 <clarkb> #link https://review.openstack.org/581214 Anomaly Detection in CI Logs
19:13:59 <clarkb> I don't think this one is ready for approval, but feedback would be good
19:14:22 <fungi> Linkid also has a wip spec up about hosting a video hosting/streaming service
19:14:36 <Linkid> hi
19:14:37 <Linkid> yes :)
19:14:47 <fungi> which has some interesting tie-ins to the containerization of services work
19:14:56 <Linkid> https://review.openstack.org/#/c/625450
19:15:07 <clarkb> #link https://review.openstack.org/#/c/625450/ video hosting/streaming spec
19:15:13 <Linkid> ^ the spec about video hosting service
19:15:14 <fungi> a good one for those who have opinions on what direction we're going in the service container space
19:15:47 <corvus> cool, i'll review that soon -- i have learned a lot about that lately :)
19:16:19 <clarkb> #topic Priority Efforts
19:16:30 <clarkb> #topic Update Config Management
19:16:56 <clarkb> there has been a bunch of progress and firefighting around ansible and docker and puppet in the last week
19:17:36 <clarkb> Good news is the bootstrapping for docker work is now merged (as of a couple hours ago), we have a script to generate our static inventory too
19:18:04 <corvus> oh great! which change was that?
19:18:05 <clarkb> Keep in mind the static inventory generation will emit an inventory with our magnum nodes in it, we don't want to use an inventory with those nodes in it, though ansible should fail to ssh into them
19:18:33 <clarkb> #link https://review.openstack.org/#/c/622964/ Inventory generation script
19:18:37 <corvus> if we apply deltas to config management, we should be able to review that
19:18:48 <corvus> so we can -1 changes which add magnum nodes :)
19:18:52 <clarkb> ya, mostly a heads up that you'll want to edit the inventory that is generated
19:19:12 <clarkb> but that should be easier than trying to scan all the clouds by hand
19:19:22 <corvus> ok https://review.openstack.org/605585 is the docker change
19:19:53 <clarkb> I've also started moving the puppet-4 work wioth futureparser enablement forward. Since chances are we'll be running puppet on various services for a while. Next up on the list are the leasticsearch nodes
19:20:02 <clarkb> thank you cmurphy for that stack of chagnes. I'm trying to get through them :)
19:20:24 <clarkb> That did expose that our groups.yaml file is a list of globs not regexes
19:20:51 <clarkb> we should all be careful of that when making changes to the groups.yaml file. + matches a literal + for example and * doesn't apply to the previous character
19:20:53 <fungi> seems we need to be periodically reminded of that
19:21:06 <fungi> easy to miss when reviewing
19:21:16 <clarkb> We also discovered a really fun ansible bug
19:21:31 <clarkb> #link https://github.com/ansible/ansible/issues/49969 Ansible bug that crashes current play and returns success
19:22:06 <clarkb> I've worked around ^ in our roles so it no longer affects us, but another warning that nested sequences of include_tasks results in sad ansible that reports happyness
19:22:15 <clarkb> and upstream is working to fix it which is good
19:22:48 <clarkb> Anything else on this effort?
19:22:50 <cmurphy> thanks clarkb
19:24:10 <clarkb> #topic Storyboard
19:24:26 <clarkb> As noted above there is the storyboard attachments spec up for approval this week
19:24:50 <clarkb> fungi has the votes necessary to move foward on db migratiosn for storyboard
19:24:51 <fungi> aside from the spec up for review and me making poor progress on the mysql move, no news i'm aware of
19:25:10 <diablo_rojo> No news besides the spec and the migration of the db really
19:25:15 <diablo_rojo> its been pretty slow going
19:25:16 <clarkb> ok
19:25:28 <clarkb> it is that time of year where everything seems to turn into molasses
19:25:40 <diablo_rojo> Indeed.
19:25:44 <clarkb> #topic OpenDev
19:25:51 <clarkb> #link https://review.openstack.org/#/c/625671/ Publish early opendev web content
19:26:18 <clarkb> Looks like this was just merged? still no website though so we will want to double check on that after hte meeting
19:26:43 <clarkb> In addition to that we now have the ability to host mailing lists at lists.opendev.org
19:27:24 <clarkb> Sounds like corvus has been busy POCing the repo hosting side of things
19:27:38 <clarkb> Looking like early next year might be a sprint to actually hosting things?
19:27:52 <clarkb> very exciting
19:27:58 <corvus> the poc i linked is running on a 3-node k8s cluster with a shared cephfs
19:28:01 <fungi> a group of people collaborating around special-purpose virtual machine monitors in rust are our first takers on lists.opendev.org
19:28:25 <fungi> for anyone wondering how obscure this was going to get right off the bat ;)
19:28:59 <corvus> mordred and i should be able to push up ansible changes to drive a gitea that way soon
19:29:20 <clarkb> corvus: is that so you can have gerrit replicate as the writer then have gitea mount the filesytem read only in a distributed manner?
19:30:12 <corvus> clarkb: yes -- but moreover, we would push to gitea in the normal way, so all of its hooks are triggered (so it records and publishes events, etc).  but the main thing is that should let us use other features in the future if we want (eg wiki).
19:30:39 <clarkb> can gitea safely do that with multiple instances writing to the same fs?
19:30:41 <corvus> basically, it would be a real HA gitea system
19:30:44 <clarkb> (things to sort out later I guess)
19:30:51 <corvus> clarkb: yes
19:30:55 <clarkb> neat
19:31:06 <corvus> i think it would be our first stateful ha system
19:31:50 <corvus> the cephfs-in-k8s thing (rook) is pretty neat too.  basically we can have all the components HA running inside k8s.  it's what k8s was actually designed to do :)
19:32:28 <fungi> from what i understand of the wiki implementation so far i'm not sure it's actually a fit for some of our current wiki.o.o use cases due to needing explicit editor rights granted from repository owners, but it's possible i'm overlooking what makes it possible to let anyone who authenticates create and edit pages. that said, the ability to drop codesearch.o.o is compelling on its own
19:32:29 <clarkb> ya the one gotcha with ceph run in that manner is that osds are tied to disk so you can't just replace hosts (have to do it in a rolling fashion with intermediate recovery points)
19:33:04 <corvus> clarkb: yep.  but we can handle k8s node failures.
19:33:05 <clarkb> whereas stateless k8s workload you can just move around all you want
19:33:27 <corvus> fungi: yeah, i'm waving hands on the wiki thing right now.  i haven't looked into it at all
19:33:44 <corvus> fungi: mostly i'm using it as a use case for "let's not back ourselves into a corner so we can't do it later if we want"
19:34:11 <corvus> oh shoot, i forgot to turn on code searching in the poc
19:34:14 <corvus> i'll do that after the meeting
19:34:31 <clarkb> alright anything else on OpenDev before we move to the general topics list?
19:35:43 <clarkb> #topic General Topics
19:36:13 <clarkb> ianw volunteered to set up a shared github account that we can use to administer our github things. This has been done and I've confirmed that I can log in to the account
19:36:19 <clarkb> #link https://review.openstack.org/#/c/624531/ ianw's docs update
19:36:33 <clarkb> I think ianw is looking for consensus on ^ before changing the setup in github itself
19:37:05 <ianw> we've got 2 votes ... i'll give it to say EOD and then i'd like to give the user permissions (and remove my own personal ones)
19:37:52 <clarkb> This simplifies new root onboarding and gets us away from using our personal accounts for this 9which we didn't want to be doing anyway)
19:38:13 <corvus> sounds good to me; i'll try it out and +2 if no issues
19:39:14 <fungi> thanks ianw!
19:39:52 <clarkb> Next on the list started as my list of changes I'd like to see merged before I am not around a computer. But they've since merged yesterday
19:39:54 <corvus> i do wonder if the username is accurate :)
19:40:36 <clarkb> since they have merged I'll go ahead and open it to anyone else to point out changes that would be good to have eyeballs on them before rooters are afk
19:41:27 <clarkb> maybe ansible lint pinning if we end up needing that
19:41:32 <clarkb> though I'm not yet aware of needing that
19:42:19 <clarkb> in any case if you spot important bug fixes please call them out so we can try to get fixes in prior to holidays
19:42:27 <clarkb> And now on to the docker mirroring issues
19:42:48 <clarkb> I'll admit I'm not completely sure I understand what the problem is here. I thought this was a known issue and is why we mirror docker on ports 8081 and 8082
19:43:08 <clarkb> Basically infra has addressed it by having mirrors that handle both cases and jobs need to know which to use based on their version of docker
19:43:27 <fungi> are there still issues? everyone i saw follow up said switching to the port 8082 mirror solved it for them
19:43:52 <clarkb> frickler: mordred corvus ^ do you know?
19:44:09 <frickler> it seems the patch has resolved things for us
19:44:14 <corvus> i don't know anything about dockerhub :(
19:44:31 <clarkb> frickler: have a link to the patch so we can record here if others run into the same issue it may serve as a good template bug fix
19:44:36 <frickler> and mgoddard also confirm things working for them
19:44:38 <fungi> s/mirror/proxy/
19:45:21 <frickler> #link  https://review.openstack.org/625596 Switch to using the pathless :8082 variant of our mirror unconditionally
19:46:04 <frickler> there is a bit of follow-up because our openstack specific mirror setup shouldn't be known to zuul-jobs
19:46:15 <clarkb> basically newer docker wants things hosted at / and so you can't tell it to talk to host.foo.com:8081/other/root/path
19:46:15 <corvus> i think that means we can drop 8081
19:46:38 <frickler> I think there were some projects that might still use 8081
19:46:39 <clarkb> so we have a host.foo.com:8082/ proxy and have kept a host.foo.com:8081/other/path for backward compat
19:46:54 <clarkb> I want to say pabelanger tested older docker against the 8082 one and it didn't work there
19:46:57 <corvus> frickler: without using that role?
19:47:00 <fungi> and part of what brought this up is that ubuntu switched to new enough docker to need the new protocol?
19:47:15 <clarkb> corvus: ya tripleo does their own mirror/proxy setup
19:47:37 <corvus> clarkb: i thought tripleo used the new url too?
19:48:14 <frickler> corvus: like this http://git.openstack.org/cgit/openstack/kayobe/tree/playbooks/kayobe-overcloud-base/overrides.yml.j2
19:48:21 <corvus> it's not important i guess; we can check the logs later
19:48:30 <clarkb> corvus: they use the new v2 api on top of the 8081 proxy because they don't actually use docker aiui
19:48:34 <clarkb> corvus: but they don't use that role
19:48:38 <clarkb> its weird
19:48:59 <clarkb> in any case it sounds like we've confirmed that 8082 does fix things and that we should start pushing people towards that address so that we can turn off 8081
19:49:11 <clarkb> probably worth a -discuss email thread for that
19:49:17 <frickler> fungi: yes, ubuntu backported a new docker into xenial-updates
19:49:23 <clarkb> and we can use apache logs to confirm that people have moved
19:50:04 <fungi> right, seems like in that case anyone using the port 8081 proxy even from xenial is broken now
19:50:26 <fungi> so odds are it'll no longer be needed but we can check apache access logs
19:50:35 <clarkb> fungi: ya, the weird one is tripleo beacuse they use 8081 wiht the new v2 url api aiui. And I think they get away with it because their client isn't the docker client
19:50:40 <frickler> except they install docker from somewhere else than ubuntu repos
19:50:45 <clarkb> so fi we switch tripleo then likely everyone else will be on 8082
19:50:50 <fungi> k
19:51:22 <clarkb> #topic Open Discussion
19:51:32 <clarkb> Anything else before we get ready to call it a year?
19:53:16 <fungi> i hope everyone enjoys whatever end-of-year festivities they prefer
19:53:38 <corvus> i'm celebrating the 8 days of k8s
19:53:45 <fungi> hah
19:53:54 <fungi> ubernete
19:54:23 <clarkb> as mentioned earlier starting tomorrow I expect to not be around much. I've got to pack for early thursday flight (and this time with kids which makes logistics a million times more complicated)
19:55:34 <corvus> i should be around this week.  not so much next.
19:55:55 <fungi> i'm around all this week and intermittently the next
19:56:05 <fungi> and then around most of the following weel again
19:56:09 <fungi> week too
19:56:10 <ianw> fyi i'm pto from EOD today till 07 jan
19:56:20 <fungi> bold!
19:56:24 <fungi> i like your style
19:56:46 <clarkb> everyone should enjoy the time they are taking off. I expect that most of openstack and related acitivities will as well
19:57:01 <clarkb> we should remind the release team to not make any releases next week :P
19:57:26 <fungi> heh
19:57:41 <clarkb> fungi: I still rememebr the year it was pbr or something made a release on the 24th and broke everything for a few days
19:57:56 <clarkb> also the year setuptools and pip released weekend before christmas or was it weekend of new years
19:58:27 <fungi> oh, yeah, fun times!
19:58:39 <clarkb> Alright sounds like that is it. We'll see you all here on the 8th of January. Until then happy/merry $holiday
19:58:44 <clarkb> #endmeeting