19:01:05 #startmeeting infra 19:01:06 Meeting started Tue Mar 5 19:01:05 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:10 The meeting name has been set to 'infra' 19:01:18 #link http://lists.openstack.org/pipermail/openstack-infra/2019-March/006291.html 19:01:40 #topic Announcements 19:02:20 The OpenStack TC election is happening right now. Those of you that have contributed to the infra project in the last year or so should have received a personal ballot in your email inbox. You have until 23:45UTC today to vote if you haven't yet 19:03:10 #link https://docs.openstack.org/infra/system-config/signing.html#attestation sign the Train release key 19:03:33 #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xcdc08088c3cb45a9be08332b2354069e5b504663&fingerprint=on OpenStack Infra (Train Cycle) artifact signing key 19:03:33 fungi has set up a train release gpg key (thank you!) and those of us with root access should follow the steps in the link above to sign it 19:04:05 That is on my todo list for post meeting so hopefully I'll get that done soon. Something to do while waiting for afs to replicate a not small volume :) 19:04:15 #link https://www.openstack.org/summit/denver-2019/call-for-presentations Submit Forum session ideas now 19:04:28 o/ 19:04:37 It is also that time of year where we plan to get together and talk about designing software/services 19:04:57 If you've got topics for that feel free to submit them 19:06:24 #topic Actions from last meeting 19:06:31 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-26-19.01.txt minutes from last meeting 19:06:50 I set an everyone action to sign up for gitea story tasks. I think that has gone reasonably well and we are making progress on the gitea front 19:07:06 yep, all tasks accounted for 19:07:07 Lets bring that up during opendev discussion though 19:07:13 corvus: yay 19:07:30 #topic Specs Approval 19:07:51 We also merged the letsencrypt spec after reviews last week. THank you everyone for that 19:08:05 Mostly just mentioning it here in case you missed it. 19:08:19 #topic Priority Efforts 19:08:52 #topic Storyboard 19:09:25 I ran through running storyboard tests locally and pushed an update to the storyboard docs that appear to have merged 19:09:30 Been slowly getting SotK's attachment patches through 19:09:56 Also still being innundated with prospective outreachy interns 19:09:59 I did that in support of making it easier for outreachy applicants. Hopefully they find it useful 19:10:07 yeah, mordred added an alternative solution relying on swift tempurls which will hopefully solve one of the security risks we missed with the previous client auth idea 19:10:25 cool sounds like good progress on the attachments implementation then 19:10:54 fungi: oh - crappit - I need to finish that / follow up on those patches 19:11:04 fungi: thanks for the reminder 19:11:04 for outreachy are there specific tasks/deadlines we should be aware of as a "hosting project" (I don't know what the actual term is) 19:11:24 the outreachy hopefuls are also burning through our new-contributor-friendly stories at am amazing pace 19:11:28 (fwiw, I've been saying "tempurl" when I mean "formsubmit" - but yes) 19:11:35 I think the application proposals for interns closes soon. 19:11:41 oh, thanks for the clarification mordred 19:11:45 Not sure what day we need to pick an intern by though 19:12:02 fungi, yeah that has been amazing 19:12:12 oh, related to interns but not outreachy, hogepodge also has some capstone interns from ndsu he's sticking on some storyboard todo items as well 19:12:13 I guess I will need to go triage some more soon 19:12:21 Oh yeah 19:12:35 specifically amending the migration script to handle bps as well 19:12:43 optionally 19:12:56 right, we've had a few teams request lp blueprints be migrated to stories 19:12:59 So we could migrate bps and bugs, or just one or the other 19:13:08 diablo_rojo: we probably want to do a review pass of those changes as part of the application process? (and when I say we I really mean people that know storyboard) 19:13:11 diablo_rojo: maybe after meeting let's sync on a plan for the formsubmit swift stuff 19:14:27 mordred, sounds good 19:14:38 SotK, should probably be around too for that 19:14:56 Are there other topics we should cover that aren't outreachy and the attachments implementation? 19:14:57 clarkb, yeah I have reviewed a few so far, but there are more to do if anyone can help out. 19:15:11 Placement will be using sb? Yay! 19:15:13 diablo_rojo: maybe we can tag them all with an outreachy topic? 19:15:26 diablo_rojo: that might help reviewers? not sure how difficult it is to sort them out from the other work 19:15:26 clarkb, oh yeah thats a good call. I can go and do that 19:15:30 great thanks 19:15:45 yeah, the last placement meeting talked about sb some, but ended in cdent's e-mail to the openstack-discuss ml 19:15:58 so probably doesn't need separate summarizing 19:16:52 ok lets move on then 19:17:00 #topic Update Config Management 19:17:37 We are ready to start deploying puppet 4 in places. https://review.openstack.org/#/c/630391/ Does that for our dev servers but was updated to fix an issue frickler caught before I could approve it 19:17:47 rereviews very welcome and I'm happy to approve that and watch it 19:17:58 yay 19:18:29 We continue to refine the docker-compose to deploy docker images with ansible process/tooling 19:18:41 we are running 4 services with that now? 19:18:47 corvus: ^ you probably know off the top of your head 19:19:19 gitea, haproxy, registry, preview 19:19:24 Also my changes to add a zuulcd user to bridge are merged so I think we can start with Zuul triggering CD jobs again 19:19:50 I guess so far just one chagne and we need to have a discussion about auto updating key data 19:20:08 but that is sufficient for now if we want to start splitting out config management runs from the always run every 15 minutes to run when necessary 19:20:21 i think we might be able to pull the git playbook out of base and run it on changes to projects.yaml 19:20:44 zuul-preview deployments likely another good candidate for that (run when zuul-preview updates) 19:20:57 i also added some suggestions on the authorized_keys retrieval change 19:20:59 yep, we can chain it to follow the promote job 19:21:12 fungi: thanks I'll take a look today 19:21:44 as a side note there, is there a reason we can't have it retrieve keys from zuul.opendev.org instead of zuul.openstack.org? i think that solves several of my concerns 19:22:00 fungi: we absolutely can. it was just written before it existed 19:22:04 yup that 19:22:12 cool, i thought (hoped) that might be the case 19:22:18 if that will alleviate concern I'll go ahead and make that update 19:22:34 i was having trouble putting together a working api url to do that yesterday but that was probably just my fault 19:22:57 clarkb: the other remaining concern will be easier to take care of later with letsencrypt 19:23:13 and is something we might want to consider doing for all our opendev le certs 19:23:51 though i'm not 100% sure the python requests module supports it (yet) 19:24:21 ok I'll take a look today 19:24:40 Are there other config management related changes/items that we want to go over before we zoom in on opendev/gitea? 19:25:07 none for which i'm aware 19:25:39 #topic OpenDev 19:25:47 #link https://storyboard.openstack.org/#!/story/2004627 opendev tasks 19:26:13 since last week, we're basically halfway done with the tasks needed before we are ready to actually move 19:26:38 We now have 8 gitea servers running behind haproxy hosting https://opendev.org 19:26:57 i think the most urgent thing is actually a new task not on there, which is to fix the default branch setting for the projects we're replicating now as well as new projects 19:27:14 mordred is working on that, with, presumably, clarkb and i helping 19:27:36 yup I'm happy to help on that 19:27:41 once replication is done and looks correct, i plan on starting on the last task on the list, which is the announcement email 19:28:10 i want to get that out as early as possible so folks know the plan and openstack and starlingx can make any preparations for repo moves 19:28:18 ++ 19:28:23 sounds great 19:28:31 but i also want to wait until we have a fully-functioning site to demo in the email 19:28:59 i've just updated the story to include previously missed airship domain name 19:28:59 meanwhile, we can work on the rest of the preparatory tasks 19:29:20 fungi: thx :) 19:29:22 is anyone stuck or needing help on any of their tasks? 19:30:17 i just need to carve out time to knock mine out. shouldn't take long 19:30:48 I'll go through devstack plugins looking for git://'s 19:31:04 the issues and release tabs don't look correct for some projects, do we have a plan for that? 19:31:13 issues always redirect to sb 19:31:32 corvus: is there a plan for port 9418 in the interim, to maybe forward to a proxy or something? 19:31:42 frickler: the issue there being we should link to launchpad in some cases? 19:31:43 ianw: no, it's not possible to support 19:32:07 frickler: we should be able to fix that by looking up if a project uses storyboard or not in the project.yaml yaml file I expect. Then run the fixup playbook across all the projects once we haev that logic in place 19:32:13 frickler: probably worth a new task to track that though 19:32:26 and releases look incomplete, compare https://opendev.org/openstack/designate/releases to https://github.com/openstack/designate/releases 19:32:29 yeah. 19:33:01 well - the releases tab is a thing we probably need to think about - maybe we should remove it from the ui for now until we have a good plan for it 19:33:17 frickler: looking at that almost appears to be a gitea bug? 19:33:30 the tags are available on the code screen at least 19:33:31 can someone tell me what the bug is? 19:33:41 corvus: only the rc releases seem to be listed 19:34:01 ah 19:34:02 well rc and beta. None of the actual releases are listed 19:34:03 well - the bug for ME is that I don't want us hosting links to things people shouldn't ever download 19:34:08 mordred: yeah, i think you were... that. yes. 19:34:12 its not consistent it seems, but e.g. for designate 7.0.0 seems to be missing 19:34:17 so let's just drop the releases page 19:34:20 yeah 19:34:35 I think it could be a nice thing to do something with at some point to have releases point to our actual releases 19:34:35 i'll make new tasks for all of these things 19:34:36 mordred: the concern being the tarball there isn't a valid python sdist? I agree that is a proble mand removing the tab would be good 19:34:37 makes sense to me 19:34:37 but that's more work 19:34:43 clarkb: yes. that is the problem 19:34:58 same bug as github's ui has - a git export isn't always a valid source tarball 19:35:45 this is good feedback and godo for us to catch these things before we announce it more widely :) 19:36:00 yep :) 19:36:47 Anything else opendev or gitea related? If not we'll move on to our general topics list 19:36:58 story updated with 3 new tasks, 2 of which are unassigned 19:37:45 clarkb: none from me 19:38:13 #topic General Topics 19:38:28 #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup Trusty Server Upgrades 19:38:37 This is mostly a status update and thank you to those helping with this 19:38:51 I've written down an afs fileserver upgrade plan on that etherpad (and am currently executing step 1) 19:39:07 if you have a chance to review that the input I've received so far has been helpful 19:39:19 And once again if you are able to upgrade a server or two please volunteer. 19:39:37 As for sprinting on this I haven't heard any responses to my email about that 19:39:53 For my schedule March 11 and 12 work best (next monday and tuesday) 19:40:13 so I'm probably going to go ahead and scribble that in as the sprint days and I'll see you there if you can make it? 19:40:57 Next up is ianw's backups topic 19:41:14 ianw seems we have some cleanup to do on the backup erver? 19:41:34 the main problem is : /dev/mapper/main-backups 3.0T 2.8T 579M 100% /opt/backups 19:42:48 now we have 3tb in "old" volumes, which we decided to clean up "in a few months" @ http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-10-31-19.01.log.html#l-189 19:43:16 ianw: I would be on board with cleaning that up by deleting the contents of those volumes, but then repurposing the voluming as the target for new backups 19:43:25 then we can do similar cleanup with the current volumes in the future? 19:43:31 i actually had a thought that we document our process here as periodically keeping the old volume, and swapping 19:43:42 ianw: ++ 19:43:50 clarkb: cool, i think we said the same thing :) 19:43:55 since bup is append only having a formal rotation process would be good 19:43:55 i like it 19:44:15 ianw: is this something you'll be able to drive? or do we need volunteer(s)? 19:44:25 clarkb: i can take an action item to do that 19:44:37 #action ianw rotate backup volumes on backup server 19:44:39 ianw: thanks 19:45:07 yeah, i didn't know if we wanted to investigate some sort of automated pruning 19:45:26 Not to completely derail, but at times I wonder if borgbackup would be helpful here. I've been using it for personal backups and my understanding is it can do append only backups with pruning 19:45:35 i wonder if we could abuse logrotate to do it 19:45:39 personally, i feel like some custom script has a non-trivial chance of wiping out much more than it intends 19:46:11 I don't think you can prune bup backups 19:46:27 at least not in bup itself 19:46:45 no, but we could do some sort of automated repo moving type thing 19:46:45 yeah, i guess rotating with retention means losing deduplication of the rotated copy 19:47:02 ianw: ya we could move the dest repo out of the way and replace it with a new empty target 19:47:08 then after X time delete the moved repo 19:47:09 it's probably spec worthy, maybe the whole backups process could do with a re-look in the opendev world 19:47:17 ++ 19:47:31 well, pruning with bup is experimental 19:47:35 corvus: ah 19:47:44 anyway, i won't sign up to take on a full overhaul :) but just flagging it for now 19:47:59 ianw: ya the earlier fix should give us breathing room to think it through 19:48:24 the other quick thing i noticed was we're not backing up stats on graphite.opendev.org (previously openstack.org) 19:48:30 fwiw I'm likely to suggest borgbackup if we intend to do a big overhaul as I've really enjoyed working with it. You can fuse mount a backup to copy files and it encrypts at the source so you don't have to trust your remote :) 19:48:33 not sure if that is a bug or feature 19:49:01 i don't think we need to back up stats 19:49:02 ianw: I'm not sure how important it is to backup that data 19:49:13 its nice to have data that we can live without 19:49:32 ok, if it's a feature, that's cool :) 19:49:37 i agree with every one of the possible ways of reading clarkb's sentence :) 19:50:13 (it's nice to have (data that we can live without)) and (it's (nice to have) data that (we can live without)) 19:50:34 :) 19:50:54 Anything else backup related? 19:51:13 no, that's it for me 19:51:17 #topic Open Discussion 19:52:07 I'm going to be in Seattle on the 19th but I should arrive early enough and be settled enough to run our meeting (especially with DST change this weekend). However the week after that (26th) we are doing a mini road trip for spring break back to the seattle area and I will likely be afk 19:52:20 if anyone really wants to run the meeting in 3 weeks let me know 19:53:04 i have no other plans, so can do that if nobody else wants to 19:53:37 Also DST change in the USA (and maybe canada?) is this weekend apparently 19:53:55 one other thing regarding gitea: can we clean up the exim paniclogs to reduce the amount of daily mails? 19:54:05 This means that our meeting will happen an hour later in the day that it happened today for those of you in the usa 19:54:49 frickler: i think it's a more general bootstrapping problem. we might want to wipe the exim paniclog if present at the end of launch-node.py 19:55:13 clarkb: so the meeting won't stay at 19:00 UTC? 19:55:22 frickler: oh sorry it stays at 1900UTC 19:55:25 pretty much every new server we've launched has a paniclog from (it appears) when exim configuraiton was first getting applied 19:55:30 It is an hour later in the day relative to my local timezone 19:55:34 ah, I read it the wrong way around 19:55:57 frickler: sorry I wasn't very clear on that. For pacific time meeting was at 11am today will be Noon next week 19:56:01 crazy folk like me who set their clocks to utc don't need to worry ;) 19:56:35 instead i get to deal with fixing the offset on all the non-community meetings in my remind file 19:56:37 my irc client's clock is utc 19:57:00 my irc client's clock is utc - and I put the infra meeting into my calendar using the iceland timezone so it just works :) 19:57:30 i probably should tag the recurring meetings in my remind file which *don't* follow utc and then just script their time changes accordingly 19:58:01 the EU might be stopping switching time zone soon ... maybe 19:58:10 https://review.openstack.org/#/c/639454/ is an example of a chagne that does post launch launch tasks 19:58:23 if anyone wants to do the cleanup exim paniclog change ^ is a good place to start probably 19:58:38 i also voted to recommend that my state legislature consider beginning a process to investigate putting together a proposal to petition the us commerce department to allow us to stop switching between pst/pdt 19:58:43 clarkb: thanks, we can probably just tack on a `rm -rf /var/log/exim4/paniclog` and call it a day 19:58:43 so i'm *very* proactive on this issue. 19:58:51 corvus: nice 19:59:02 oregon has a bill that gets rejected every year to ditch dst (or go to dst fulltime) 19:59:16 And we are just about at time. Thank you everyone 19:59:19 #endmeeting