19:00:29 #startmeeting infra 19:00:30 Meeting started Tue Jun 18 19:00:29 2019 UTC and is due to finish in 60 minutes. The chair is ianw. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:31 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:33 The meeting name has been set to 'infra' 19:01:02 the agenda for today is @ 19:01:03 #link http://lists.openstack.org/pipermail/openstack-infra/2019-June/006403.html 19:01:15 #topic Announcements 19:01:53 clarkb obviously doing something more exciting than running this meeting :) 19:02:12 i don't think we have anything else to announce 19:02:45 oh, one thing 19:02:54 #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006888.html Long overdue cleanups of Zuulv2 compatibility base configs 19:03:03 \o/ 19:03:19 zuul-cloner will no longer be a thing ? :) 19:03:32 #info zuul-cloner shim and bindep fallback are being removed from non-legacy abstract jobs on Monday, June 24 19:04:02 gonna follow up to that announcement on the ml with a reminder 19:04:32 and put on my flame retardant underpants when i get up monday 19:04:32 ++ 19:05:37 #topic Actions from last meeting 19:05:49 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-11-19.01.html 19:06:06 corvus: work with jroll to get openstack github org self managed 19:06:17 derp 19:06:22 sorry 19:06:29 did not happen :( 19:06:38 * mordred also supposed to make opendevadmin user - will do after meeting 19:06:47 should we re-action or is it on todo list? 19:07:10 mordred: set up github opendevadmin account for the opendevorg org and zuul app 19:07:30 let's re-action it, not only as a safetly valve in case i forget again, but also, it seems like something i should report back on regardless 19:07:45 #action corvus work with jroll to get openstack github org self managed 19:07:53 same with admin account 19:08:09 #action mordred set up github opendevadmin account for the opendevorg org and zuul app 19:08:40 ok that was it for that 19:08:57 #topic Priority Efforts 19:09:18 #topic Update Config Management 19:09:48 can talk about specific mirror stuff later 19:10:45 did we have any efforts underway on the config management side? 19:10:58 i guess it's down mostly to cleanup now 19:11:09 i will have to check out the builders -- note i merged https://review.opendev.org/#/c/665014/ to stop building the control plane images 19:11:32 and some of this is intertwined with the trusty upgrades as well as the mirror updates you noted 19:11:37 last night i noticed that nb01 had about 250 threads which i don't think was right, errors trying to delete in-use images 19:11:57 that definitely does not sound right 19:12:07 it doesn't sound ideal at the very least 19:12:39 anyway will investigate more, i did py-bt dump all them but it wasn't pointing to a smoking gun inside nodepool-builder/openstacksdk 19:13:37 anything else? 19:14:02 #topic OpenDev 19:14:38 For the thread about ara 1.0 on openstack-infra 19:15:31 some pieces of the puzzle are falling in place -- we have an implementation of the sqlite middleware working for 1.0 that will land soon 19:15:38 #undo 19:15:39 woot 19:15:39 Removing item from minutes: #topic OpenDev 19:15:56 I somehow read that as open floor 19:16:02 sorry for sidetracking 19:16:34 oh, cool. yeah, as noted by corvus i guess with swift logs we still have an issue, but at least getting status quo to 1.0 will be good 19:16:44 there are some question marks around swift, yeah 19:17:17 basically 19:17:39 the equivalent middleware in 1.0 will give us arbitrary API endpoints to every job results 19:17:53 there are question marks around middleware too.... our plan is to stop running it and rely entirely on statig generation 19:17:57 and ara-web is able to connect to any of those arbitrary endpoints 19:18:37 that plan is ready to go at a moments notice (for example, the next time our filesystem is corrupted) 19:19:08 we're overdue for one of those events 19:19:34 as far as full static generation goes, I have a half working proof of concept but we need some changes under the hood for it to work properly 19:19:58 like, I have the actual html of the pages as if you would be browsing them but it still attempts to connect to the API 19:20:14 this is tracked here: https://github.com/ansible-community/ara-web/issues/11 19:20:57 #link https://github.com/ansible-community/ara-web/issues/11 19:21:00 cool -- i think i'd suggest that we not spend too much effort on doing ara 1.0 middleware in opendev given that we might switch to static generation at any time 19:21:10 seconded 19:21:35 we can always pin ara<1 until we switch to object storage 19:21:42 works for me 19:23:00 i suggested on list that maybe this is suitable for a single service that we sent the results to in post in the job, but i'll read through the static generation stuff to understand more too 19:23:26 i didn't think react was amenable to that, but if it is, great! 19:24:06 one other little thing is that i discussed having a vexxhost backup server with mnaser, and we agreed it was ok, so that's on my todo to bring up, and get to comments left in the review for ansible backups left last week 19:24:26 ok we should move on 19:24:33 #topic OpenDev 19:24:49 i guess this was mostly the github mgmt bits which are in flight 19:25:26 any other particular updates? 19:25:33 on that note 19:25:53 dmsimard brought up in #zuul that we still have stale mirrors of things in the openstack-infra org on gh 19:26:01 things which didn't move to the openstack org 19:26:24 worth deciding (not necessarily in-meeting) what we want to do with those 19:26:29 I vote that we push up a retired commit with a pointer to opendev.org and then archive each of them 19:26:30 i'd guess there's quite a lot of openstack-infra/ repos in that state 19:26:36 ianw: where can one see the github mgmt bits? I haven't really been following along as much as I would like 19:26:39 but also agree, we don't have to agree to a plan of action here 19:26:44 whether we push readme files to all of them or delete them and update the org description or something else entirely 19:27:38 * mordred can probably handle doing the retire/archive before being AFK again, if people are comfortable with that approach 19:27:46 pabelanger: sorry, yes very abstract, i was referring to the discussion started last week @ http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-11-19.01.log.html#l-105 19:28:08 ianw: thanks 19:28:11 mordred: +1 19:28:11 i'm most ni favor of whatever someone else volunteers to do. i don't have strong feelings about our lingering repository presence on github 19:28:33 cool. I'll take a stab at $something - and will either accomplish it or give up 19:28:40 other than agreeing that stale git repositories there is probably the least desirable option 19:28:45 fungi: ++ 19:29:09 ok, that sounds like 19:29:15 thanks mordred! and thanks dmsimard for bringing it up! 19:29:34 #action mordred look at stale github.com/openstack-infra repos and come up with retirement/removal plan/changes 19:29:38 ok? 19:29:45 where are we keeping the rename mapping file from the Great Rename? I imagine it'll make automating the retire commits easier 19:29:47 (or jdi) 19:29:49 ianw: ++ 19:30:21 there were multiple phases of renames 19:30:34 may be easier to build a list by querying the gh api org object 19:30:55 anything in the old openstack-infra org, retire 19:31:27 well, we have a file from the most recent renames 19:31:29 well - I was mostly thinking of getting the openstack-infra/zuul -> zuul/zuul but openstack-infra/puppet-zuul -> opendev/puppet-zuul mapping 19:31:39 and i bet we have the file from the great rename too 19:31:44 those 2 should be all that's needed 19:31:49 I agree - it should be applied to all things in gh openstack-infra org 19:31:53 oh, right, that file. i had already forgotten we put it all in there back to the migration batch 19:31:55 corvus: yah 19:32:07 mordred: I might have some code to share about adding 'archived' support to managed-projects, that is what we are using in zuul.a.c, and works well :) 19:32:22 #link https://opendev.org/opendev/project-config/src/branch/master/renames 19:32:25 pabelanger: sweet! I'll totally borrow that from you 19:32:37 https://github.com/ansible/project-config/commit/bb2d4d18ed79ef68e6f8b22b08cecd0a14f6faf5 19:32:39 fungi: AWESOME - thank you 19:32:49 "unmanage-projects" 19:32:50 happy to push that up 19:33:06 pabelanger: that's nice and easy to copy 19:33:20 pabelanger: I'll likely write a one-off script for this, so that's good enough for me 19:34:31 ack 19:34:33 (it will almost certainly be a very ugly script) 19:35:04 #topic Storyboard 19:35:23 storyboard-api releases on pypi are a thing now 19:35:38 we released 1.0.0 last week and 1.0.1 over the weekend 19:35:56 or actually yesterday morning my time 19:36:16 neat 19:36:31 over the weekend, logins were broken because of the distname change from storyboard to storyboard-api 19:36:53 for whatever reason, the auth codepath includes asking pbr for some dist info 19:37:05 o_O 19:37:06 and we missed updatnig the string being passed into it 19:37:44 that along with a very annoying but infrequent db lock contention bug were fixed in 1.0.1 19:38:11 fungi: i think take a don't ask, don't tell policy with that :) 19:38:32 (the pbr stuff) 19:38:37 indeed 19:38:55 i mean, we can pickaxe the history, but it likely will take us back to commit #1 19:39:11 it's probably my fault 19:39:18 also the karbor team changed their minds and decided they did want their lp bugs imported after all, so i did that yesterday 19:39:55 ok, anything else? 19:39:58 i don't know of any other pressing news, but maybe SotK or diablo_rojo are around? 19:40:10 if not, we can surely move on 19:40:39 #topic General Topics 19:40:48 trusty upgrade 19:40:57 #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup 19:41:12 still poking at the wiki-dev02 xenial build as tmie allows 19:41:21 and then i think it's really just status.openstack.org ? 19:41:27 I continue to not actually do the status upgrade 19:41:39 but I'm totally enjoying continuing to not actually do it 19:41:47 I got nothing pressing 19:42:15 the current snag is that something like 2/3 of our ~40 extensions and our half a dozen skins can't be cloned by the vcsrepo module and their .git ends up being a file instead of a directory, with just a single line like "gitdir: ../../.git/modules/extensions/Renameuser" 19:42:25 not sure if that means it's some submodule magic or what 19:42:59 (does that pattern look familiar to anyone?) 19:43:38 fungi: no - but I think your reasoning that leads you to suspect submodules seems likely 19:44:03 we're cloning from https://gerrit.wikimedia.org/r/p/mediawiki/${type}s/${name}.git 19:44:09 no but i seem to remember having some issue with vcsrepo that required a much later version ... trying to remember what that was ... i think with ask.o.o ... 19:44:13 #link https://opendev.org/opendev/puppet-mediawiki/src/branch/master/manifests/extension.pp#L13 19:44:59 https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git is also getting us a clone like that 19:45:21 but if i got clone it manually i'm getting an actual repo 19:45:28 s/got/git/ 19:45:29 oh my. 19:45:38 got git? 19:45:44 anyway, not going to solve it right now, bit anyone who has ideas, i'm all ears 19:46:00 ianw's vcsrepo version concern is definitely worth digging into 19:46:20 we can probably move on as there's more on the agenda 19:46:42 ok 19:46:58 yeah i wanted to update on the opendev.org mirror situation 19:47:11 several things in flight there 19:47:21 in short, if we could review 19:47:32 #link https://review.opendev.org/#/q/status:open+branch:master+topic:kafs 19:48:00 starred 19:48:40 basically the openafs version shipping with bionic (1.8.0~pre5) is buggy; so we either use our own built later versions, or something else like kafs 19:49:20 kafs, for those not involved, is the inbuilt kernel support for afs. so we don't have to build any modules with dkms etc etc like with openafs. the only issue is you need a very recent kernel 19:49:44 like, top of tree master kernel (5.2-rcX) 19:49:49 from what we can tell so far, one so new it's not released yet ;) 19:50:18 so that is obviously not without issue either 19:50:45 it is interesting to be on the bleeding edge of afs development. 19:50:54 we have mirror.iad.opendev.org running this, currently setup by hand, but the reviews in the above topic make it all fairly nicely ansible controlled 19:51:10 i'm just happy that there is still a bleednig edge for afs development 19:51:24 we also have mirror.dfw.opendev.org running upstream 1.8.3 openafs taken from packages i put in our ppa 19:51:49 afaict, that has been running fine for several days now 19:53:09 * mordred is supportive of both options 19:53:24 i feel like at this point we would want to roll out the servers with openafs 1.8.3 -- devil you know --(changes to do that in the stack above too), but opt in a few to kafs? 19:53:47 i would like to bring up a very remote to rax kafs node, to see what latency and network weirdness does to it 19:53:47 it would be nice if ubuntu could at least replace their prerelease afs package with a more updated one that isn't thoroughly broken 19:53:55 but my hopes are not high there 19:54:06 fungi: i have filed an issue ... if we know a way to push on that i'm happy to 19:54:07 (for bionic i mean) 19:54:15 the shutdown of the kafs server is suspicious, but it still could be a random unrelated incident. 19:54:15 there's a link in the reviews 19:54:37 need to look into that ... i had a little background monitoring going on too after discussion in openafs last night 19:54:45 right, i'm far from blaming kafs on an instance in rackspace spontaneously (from our perspective) going into shutdown state 19:54:46 it's also probably easier to maintain our own openafs packages rather than a kernel 19:55:22 this is true 19:55:35 though the kafs implementation has an awful lot of cool factor to it 19:55:45 and that sure counts for something 19:55:49 yeah, with 1.8.3 seeming to be stable now, i think that should be our default. but we've got great CI, and with those changes we can keep both paths in flight 19:56:10 (secretly, i'm hoping upstream notice just how cool our CI is and want to get more involved than just occasional advice ...) 19:56:19 ianw: i like both of your suggestions -- maybe start thinking about resolving the current situation with openafs and continue testing kafs in more situations, since it may be good for us long-term 19:56:42 i too see kafs as worthwhile investment in the future 19:56:54 even if we don't end up relying on it right now 19:57:28 ++ seems like we're all on about the same page :) 19:57:44 #topic Open discussion 19:57:52 any last minute review requests, etc? 19:59:06 #link https://review.opendev.org/655783 Don't hide Zuul CI comments 19:59:12 might be nice to come to a decision around that 20:00:04 jhesketh had some good suggestions 20:00:21 hide all but last seems like a nice intermediate, if the hacking required isn't too crazy 20:01:41 ok, thanks everyone, back to #openstack-infra for any other concerns! 20:01:45 #endmeeting