16:00:23 <mattmceuen> #startmeeting Airship 16:00:24 <openstack> Meeting started Tue May 21 16:00:23 2019 UTC and is due to finish in 60 minutes. The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:27 <openstack> The meeting name has been set to 'airship' 16:00:28 <mattmceuen> #topic Rollcall 16:00:32 <alexanderhughes> o/ 16:00:34 <aaronsheffield> o/ 16:00:34 <mattmceuen> Good morning / evening everyone! 16:00:38 <michael-beaver> o/ 16:00:39 <mattmceuen> https://etherpad.openstack.org/p/airship-meeting-2019-05-21 16:00:41 <evgenyl> Hi! 16:00:48 <dwalt> o/ ! 16:00:52 <mattmceuen> ^ agenda, let's give folks a couple mins to get here and add stuff to the agenda 16:00:54 <nishantkr> o/ 16:01:24 <mattmceuen> dwalt it looks like you're raising your arm in alarm - it's not a stickup, I promise 16:01:31 <arunkant> o/ 16:01:39 <roman_g> o/ 16:02:02 <alexanderhughes> if it was, wouldn't it be \o/ 16:02:18 <mattmceuen> ! \o/ ! 16:02:20 <openstack> mattmceuen: Error: "\o/" is not a valid command. 16:02:35 <kskels> o/ 16:02:54 <mattmceuen> Great turnout today, thanks for being here everyone :) 16:02:57 <mattmceuen> #topic Email notifications to mailing list for failing, periodic jobs 16:03:07 <mattmceuen> this one is dwalt's - take it away sir 16:03:28 <dwalt> lol - missed the comment above. Not alarmed, just happy to be here :) 16:04:29 <dwalt> So mattmceuen and I have been discussing some jobs we have that would be better suited as periodic jobs. e.g. chart linting against the latest helm-toolkit from osh-infra and a job that would verify our credentials for posting images and publishing images is correct 16:04:32 <dwalt> are* correct 16:05:19 <dwalt> An idea we had was emailing the results of the aforementioned jobs to a new mailing list, perhaps airship-alerts or airship-builds.. something like that 16:06:17 <dwalt> I added this to the agenda in case for a few reasons. 1: for knowledge sharing if anyone has set something up like this before (looks like there is an email template with zuul), 2: for feedback, if anyone has additional ideas for jobs or comments about the idea in general 16:06:46 <mattmceuen> So looking for prior art, here are the openstack mailing lists: http://lists.openstack.org/cgi-bin/mailman/listinfo 16:06:50 <alexanderhughes> Roman and I had a similar discussion for readthedocs post job, and the major concern we had was "spam" whatever e-mail address this comes from needs to be easily identifiable so it doesn't go to the spam box 16:06:56 <roman_g> failing docs build jobs are now e-mailing to me. 16:06:59 <dwalt> The idea is just to raise awareness about these issues before they strike us too late. For example, we recently learned we haven't published images in 20 days :o 16:07:10 <roman_g> I can set it to e-mail to the mailing list, I think 16:07:14 <dwalt> roman_g: is that directly from RTD or zuul? 16:07:21 <roman_g> that's from RTD 16:07:34 <roman_g> Zuul has no clue that RTD build has failed (or not) 16:07:45 <dwalt> do your patches make it so? 16:07:47 <roman_g> ^^^^^^^^^^^^^^ - working on it 16:07:48 <mattmceuen> I like the idea of creating a new ML for automation-oriented emails, since it may have a different audience of folks who care than the announce and discuss lists 16:07:56 <mattmceuen> but am open to other thoughts on that 16:08:13 <roman_g> no, it's just a setting to e-mail about failed builds on RTD service 16:09:00 <dwalt> I do see the release-job-failures on your link mattmceuen. Sounds like this is common practice, so that's a good thing 16:09:09 <mattmceuen> Yeah, that's the one that seemed most similar to me too 16:09:41 <dwalt> I also like the idea of a separate mailing list. Does anyone have any thoughts on that? 16:10:16 <mattmceuen> `Airship-automation`or `Airship-bot` or something 16:10:26 <dwalt> roman_g: That is definitely a good start. I would propose we make that the first thing to email the new mailing list 16:10:39 <dwalt> Until we can fully detect failures in the docs zuul job 16:11:01 <dwalt> I do like airship-automation. Or airship-integration 16:11:13 <roman_g> OK. Get new mailing list added, and I will set up RTD to report there (need to have access to airship-treasuremap RTD, Stacey holds the keys) 16:11:40 <dwalt> Another thing we can add to the list is the GitHub mirroring, I see we will be discussing that later. SO if no one has any other thoughts on this, we can get to work! 16:11:42 <dwalt> thanks roman_g 16:11:48 <mattmceuen> Cool. Sounds like this new ML will have multiple use cases and noone has any concerns? 16:12:10 <kskels> I think I have access to RTD also, I'll ping you Roman sometime tomorrow 16:12:11 <alexanderhughes> nope, good to see some automated alerting on things we can't easily see post merge of patches - changed credentials to quay (image builds), rtd etc 16:12:29 <mattmceuen> awesome. Moving on then: 16:12:33 <mattmceuen> #topic All deckhand integration gate jobs are failing 16:12:33 <roman_g> kskels: great, thank you. 16:12:42 <mattmceuen> Yes - this one is not good 16:13:08 <mattmceuen> I chatted with Dan Crank on this since he is back from vacation, and dug a little bit this morning 16:13:31 <mattmceuen> I'm able to reproduce the issue in a VM using the `integration-tests.sh` script as well in deckhand 16:14:11 <mattmceuen> deployment of the minimal cluster (in minikube) to run deckhand on top of progresses until it stalls on Keystone 16:14:57 <dwalt> how does it install Keystone? If the Airskiff job is passing, may be worthy taking the same approach 16:15:01 <mattmceuen> Keystone deployment fails with: 16:15:07 <mattmceuen> https://www.irccloud.com/pastebin/ndKQJiP6/ 16:15:33 <mattmceuen> Simultaneously to that, the API Server is getting errors like: 16:15:35 <mattmceuen> https://www.irccloud.com/pastebin/8XtzhaYE/ 16:15:56 <mattmceuen> The OSH-Infra ansible scripts dwalt - may be pinned to a different version though 16:16:18 <mattmceuen> I haven't had time to dig deeper than that, does this jog anyone's memory for a likely root cause? 16:17:15 <mattmceuen> dwalt: agree, that's a good idea. the airskiff job is already using the latest and greatest from osh-infra, so piggybacking on it might even be faster than fixing whatever's going wrong here 16:17:18 <dwalt> Nothing from me -- I was wondering about the version 16:17:41 <dwalt> If we could make the Airskiff stuff a little more resuable, I agree, that would help 16:18:08 <dwalt> it's hard to maintain the Deckhand jobs, because they're not similar to the rest 16:18:23 <mattmceuen> Ok - I will take this as a priority today unless anyone else would like to volunteer 16:19:13 <mattmceuen> Unless anything else on this, moving on: 16:19:29 <mattmceuen> #topic AirshipCTL spec is up for review 16:20:05 <mattmceuen> Just to raise awareness that jezogwza has a WIP spec up for the airship 2.0 `airshipctl` CLI: https://review.opendev.org/#/c/659808/ 16:20:36 <mattmceuen> He's asked for folks to give it a thorough going-over so we can iterate on it and get it in good shape prior to any implementation work 16:21:19 <roman_g> mattmceuen: would like to have short call w/ you on this matter. 16:21:39 <roman_g> will ping you later 16:21:45 <mattmceuen> sure thing roman_g 16:22:17 <mattmceuen> #topic Certificate update (and how to change them easily) 16:22:30 <mattmceuen> I can't see who added this -- lemko, is it yours? 16:23:13 <mattmceuen> lemko was attempting to update certificate overrides for promenade's (calico) etcd chart, and hit issues 16:23:13 <lemko> Yes it is me 16:23:33 <mattmceuen> logs: http://paste.openstack.org/show/751880/ 16:25:09 <mattmceuen> I've been told from the folks that tested password rotation that they've cert rotation has been tested too, so I believe this is either an issue with an old version of the chart lemko has deployed perhaps, or, it's a regression 16:25:23 <mattmceuen> Either way I think we need to test cert rotation using the latest charts 16:25:41 <mattmceuen> lemko would you be able to try reproducing using the latest promenade? 16:26:32 <lemko> Ok I'll try using latest promenade. Thanks 16:26:57 <lemko> I'll upgrade my environment 16:27:20 <mattmceuen> ok, thanks - let me know how that goes! 16:28:16 <mattmceuen> alright, unless anything else on cert rotation, moving on: 16:28:37 <mattmceuen> #topic GitHub Mirrors 16:28:59 <mattmceuen> last week, kskels re-set-up our github mirroring (thanks kskels!!) 16:29:19 <kskels> I did submit a first PS for treasuremap repo only for now to test it.. 16:29:19 <mattmceuen> https://github.com/airshipit/ 16:29:38 <mattmceuen> evgenyl pointed out: Should github mirrors work already? https://github.com/airshipit/treasuremap (doesn't look to be up-to-date) 16:29:50 <mattmceuen> Do you expect the sync'ing to be in effect yet kskels? 16:29:53 <kskels> I think it should start working (or failing) on the next PS that comes in 16:30:10 <mattmceuen> fantastic 16:30:13 <dwalt> The post jobs are failing http://zuul.openstack.org/builds?job_name=treasuremap-upload-git-mirror 16:30:14 <kskels> so the way it works is a POST job for merge.. 16:30:17 <kskels> but it's till neter tested 16:30:26 <kskels> so might need some tuning.. 16:30:49 <dwalt> it said something about a key, I didn't quite understand where that came into play 16:31:01 <openstackgerrit> Sean Eagan proposed airship/promenade master: Fix potential port conflict https://review.opendev.org/659369 16:31:26 <kskels> https://www.irccloud.com/pastebin/2ya1018p/ 16:31:37 <roman_g> airshipbot@github.com: Permission denied (publickey) ? 16:31:40 <kskels> yeah - I will take a look on this tomorrow once I'm back in the office 16:31:50 <kskels> unless someone can already figure this out 16:32:00 <mattmceuen> oh boy, are you on vacation today kskels? 16:32:06 <kskels> sick :( 16:32:09 <mattmceuen> :( 16:32:18 <mattmceuen> Hope you feel better dude 16:33:17 <mattmceuen> Once we get the key stuff sorted out, this'll be great. 16:33:46 <kskels> right.. we will then need to implement this for all repos. 16:33:50 <mattmceuen> Yup 16:34:12 <mattmceuen> Anything else on Github mirroring, folks? 16:34:50 <mattmceuen> #topic Treauremap-based gates capacity 16:34:55 <mattmceuen> this one is yours roman_g 16:35:25 <roman_g> Yep. There are 2 patches to limit versions.yaml updater to only Ubuntu-based images 16:36:00 <roman_g> We are probably not able to run gates x2 times, to test both ubuntu and opensuse images. 16:36:28 <roman_g> Does anyone have objection or opinion in that? 16:36:43 <roman_g> *on 16:36:56 <mattmceuen> this was discussed a bit in the ptg too 16:36:56 <evgenyl> Yep, definitely not for BM, we can return to this questions when we get multinode and aiab moved into treasuremap. 16:37:01 <dwalt> Is an Airskiff job for opensuse images still in the works? I know we discussed at PTG 16:37:30 <mattmceuen> I like your PS roman_g as it makes the updater script parameterizable (you can choose the distro to update), rather than hard coded to ubuntu, right? 16:37:52 <roman_g> mattmceuen: yes. But pipeline is hardcoded at the moment to ubuntu 16:38:04 <roman_g> *will be 16:38:44 <mattmceuen> yup - and that's fine, because opensuse isn't really planned to be used with maas-based provisioning, so an airskiff-type deployment is a better integration setup until we get ironic support in there 16:39:37 <roman_g> So... appreciate the reviews 16:39:50 <mattmceuen> dwalt I'm not sure if anyone's actively working on the opensuse airskiff setup yet 16:40:01 <roman_g> and there is a need to talk to SUSE on creation of OpenSUSE-ased treasuremap 16:40:08 <mattmceuen> roman_g: cool I added it to the review request list :) 16:40:29 <dwalt> that makes sense since the Deckhand image is not yet published. Just curious :) 16:40:40 <roman_g> arunkant: are you here? Whom to talk to? 16:41:37 <roman_g> dwalt: this is taken care of. If there is no image mathcing the filter, the first latest tag is being used 16:41:41 <mattmceuen> we may have to catch up with arunkant tomorrow 16:41:57 <dwalt> sorry roman_g, was referring to airskiff 16:42:20 <roman_g> from what I've understood, they are not aware that they potentially would need to build own gate 16:42:30 <roman_g> dwalt: np 16:42:39 <jamesgu__> roman_g: what about new gate? 16:42:56 <roman_g> jamesgu__: good you are here :) 16:43:04 <jamesgu__> distracted, but back 16:43:20 <roman_g> Treasuremap gates are running Ubuntu and will use Ubuntu-based images. 16:43:57 <roman_g> There is probably a need to have OpenSUSE-based third-party gates running from Treasuremap repo 16:44:37 <roman_g> mattmceuen, kskels any comments? 16:44:55 <mattmceuen> jamesgu_, dwalt, remind me: we were planning on running the opensuse airskiff job on the different projects, right? Or, was it planned as a periodic job? 16:45:27 <mattmceuen> I.e., run it on openstack infra 16:45:28 <dwalt> I understood it to be on each project, alongside the existing Ubuntu jobs 16:45:35 <mattmceuen> yup - that makes sense to me 16:45:54 <jamesgu__> yes, we talked the possibility of 3rd party for BM, but VM gates is not an issue, i.e., can be in Zuul 16:46:13 <mattmceuen> So one way we could do that would be to add e.g. an airskiff-opensuse site, and then update a site-level versions.yaml with overrides for opensuse images, or something 16:47:31 <jamesgu__> mattmceuen: it makes sense to me. do we need to start with a spec? sounds like you suggest we start with Airskiff gate? 16:48:07 <jamesgu__> treasuremap uplifting gate is the one that needs BM, right? 16:48:18 <roman_g> no spec is probably needed. ideally the only difference between sites should be versions.yaml 16:48:26 <mattmceuen> jamesgu_ : may be a small enough thing to design for that we don' tneed a spec - do you have anyone who could focus on that, if we discussed it in next week's meeting? 16:48:36 <evgenyl> jamesgu__: We will be able to do uplift using a virtual multinode gate, once it is moved. 16:48:55 <jamesgu__> evgeny1: that's awesome 16:49:02 <roman_g> yes, treasuremap repository. there are 3 sites defined there, one of them is BM seaworthy, others are virtual 16:49:13 <kskels> airsloop is also bare-metal 16:49:18 <jamesgu__> mattmceuen: we will ahve someone to work on that in a couple of weeks I hope. 16:49:19 <roman_g> oh, yes 16:49:27 <roman_g> airsloop is BM 16:49:49 <mattmceuen> jamesgu__ awesome -- whenever that person is able to join, let's have the conversation then so that they can be a part of it, sound good? 16:49:57 <jamesgu__> we can discuss in next week's meeting for sure to get it started 16:50:01 <mattmceuen> perfect 16:50:10 <mattmceuen> we'll plan on that 16:50:29 <mattmceuen> Ok! 16:50:35 <mattmceuen> #topic Requests for Review 16:50:42 <mattmceuen> https://review.opendev.org/#/c/656988/ - Pegleg, decrypt command bugfix 16:50:42 <mattmceuen> https://review.opendev.org/#/c/658380/ - Pegleg, base64 encoding in passphrase catalog (needed for k8s) 16:50:42 <mattmceuen> https://review.opendev.org/660429 Add tag filter parameter to the updater tool 16:50:42 <mattmceuen> https://review.opendev.org/660441 Update Jenkinsfile's to use Ubuntu-based images 16:50:42 <mattmceuen> https://review.opendev.org/#/c/659808/ - airshipctl 16:50:52 <mattmceuen> Some of these have already been discussed here today 16:51:02 <mattmceuen> Please try to give them some review today or tomorrow 16:51:15 <mattmceuen> any others with a big need for review? 16:51:51 <mattmceuen> #topic Roundtable 16:52:07 <mattmceuen> Open mic - anything else to bring up? 16:52:11 <mattmceuen> Karaoke? 16:52:18 <alexanderhughes> core reviewer updates? 16:53:07 <mattmceuen> oh! good point - I need to send out an email as well, the one-week deadline snuck up on me :) 16:53:35 <alexanderhughes> time flies on the airship 16:53:36 <mattmceuen> I'd like to welcom alexanderhughes as a core reviewer for sphyglass and pegleg, and ian-pittwood as a core reviewer for spyglass 16:53:41 <mattmceuen> it sure does :) 16:54:12 <mattmceuen> happy to have you guys' continued focus on these projects from a core reviewer perspective! I will update the necessary checkboxes today 16:54:29 <ian-pittwood> Thanks mattmceuen! 16:54:32 <alexanderhughes> thanks :) 16:54:38 <dwalt> congrats! Thanks for all the hard work, looking forward to working with you both :) 16:54:42 <roman_g> \o/ 16:55:13 <mattmceuen> \o/ ! 16:55:46 <mattmceuen> Any other items to bring up today, team? 16:56:52 <mattmceuen> in that case, have a great rest of your day / week! thanks for your time & discussion today 16:57:06 <mattmceuen> #endmeeting