22:01:12 #startmeeting zuul 22:01:12 Meeting started Mon Mar 12 22:01:12 2018 UTC and is due to finish in 60 minutes. The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:01:13 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:01:16 The meeting name has been set to 'zuul' 22:01:20 #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul 22:01:27 #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-02-19-22.01.html 22:01:32 o/ 22:01:55 #topic Action items 22:02:00 o/ 22:02:03 rbergeron to mail list with times she signed people up for after thorough research and hope for non-conflict to do zuul interviews for yayyyy, content 22:02:15 that, or something like it, happened! 22:02:27 i think zuul interviews were had at the ptg 22:02:31 i didn't see the interviews. anybody have a link? 22:02:37 and there should be stuff up on youtube 22:02:45 i haven't seen it yet, myself 22:03:02 yes saw the zuul video :) 22:03:05 i saw the jim & jesse show. captivating! 22:03:23 there was also a second one 22:03:57 and I did one for infra 22:04:00 wiht pabelanger 22:04:21 #link jim and jesse interview https://www.youtube.com/watch?v=t1Ae_gkXOiQ 22:04:53 thanks! 22:04:58 corvus: I think that's the version without the audio fix 22:05:15 There's an "edited" version with less background noises/air conditioning 22:05:25 dmsimard: oh i thought that was the fixed version 22:05:34 Maybe it is, I'll check 22:05:59 Doh, it is the fixed version, my bad 22:06:05 cool 22:06:28 i can't find others right now, i guess let's dig for them later 22:06:46 #topic Possibility to shift meeting back about two hours? (tobiash) 22:07:08 yah, I'd really like to attend the zuul meeting 22:07:27 but it's currently 23:00 for me and 0:00 in summer time 22:07:36 yuck 22:07:42 i'm all for earlier times 22:07:48 i'm free mondays at 2000z 22:07:50 i'll note that no one west of me has announced their presence at this meeting, so we may not have much in the way of objections from this group :) 22:07:56 so I'd kindly ask if it's possible to shift it to an earlier time 22:08:09 :) 22:08:09 mostly works for .au ... that will be 7am for the next little while, 6am after daylight savings changes 22:08:37 i think when we set the time we had more late-rising apac folks 22:08:40 it's still an hour later than we've been doing openstack-infra team meetings, which i'll grant is terribad for apac attendees 22:08:46 and fewer eu folks 22:09:08 is jhesketh here too? 22:09:26 that said there probably isn't a time that is good for eu, na, and apac 22:09:54 i heard someone say that TristanC is moving back to the americas, so if that's the case he's one less apac-timer 22:10:10 clarkb: right. Maybe we could post the meeting notes to the mailing list ? 22:10:23 i think in the entire history of openstack, the best time we've come up with is 20:00 +/- one hour -- it seems to be about the sweet spot in terms of equalizing the pain. 22:10:48 discussing the meeting time change proposal on zuul-discuss would probably be a good way to get more inclusive feedback from other timezones 22:11:13 that's why we had the infra, tc, and project-wide meetings on those times 22:11:40 fungi: definitely -- i think that's the next step 22:12:37 while we're thinking about it, do we want to stick with monday, or move it to tuesday? there's no longer an openstack conflict at 20:00... :| 22:13:02 I'm fine with both days 22:13:04 moving it to tuesday would certainly simplify my meeting line up 22:13:09 the advantage to moving it is that we can be adjacent to the openstack-infra meeting, and we don't get canceled for holidays as much. 22:13:10 but thats not a major concern 22:13:21 the advantage to keeping it on monday is that we get to cancel it more often for holidays :) 22:14:25 i wouldn't like tuesday, but that's purely for selfish reasons 22:15:13 (all my other meetings are on tuesday, so this would mean i'd get to go back to just writing off one day for meetings, and i'd "gain" a workday :) 22:15:14 blarg, I missed the start of the meeting. oops. 22:15:47 jlk: a surprisingly on-topic comment! :) 22:15:52 okay, let's solicit feedback from the mailing list on both the date and time changes.... 22:15:58 tobiash: you want to send a mail, or should i? 22:15:58 i'd be fine with tuesday at this point. after the tc went long enough not using that slot we officially removed it from the schedule 22:16:04 heh I even asked about it before hand. My calendar entry didn't have an alarm associated with it. 22:16:11 I can send it tomorrow 22:16:31 (i'm also fine with either day, fwiw) 22:16:51 anything else on this topic? 22:17:45 #action tobiash send a proposal to the mailing list to discuss moving meeting to 20:00 or thereabouts 22:17:53 #topic Security notification process (corvus) 22:18:32 we're about to send out a notification about a vulnerability in zuul we just fixed 22:18:48 good thing we didn't release v3 already! ;) 22:19:19 fortunately, it's not bad, as it does not seem possible to exploit in the standard configuration 22:19:53 but it has pointed out we don't really have a documented process for security issues 22:20:15 yes, i think, before we _do_ have our big v3 release splash, it would be swell to have a clear process for reporting and announcing them 22:20:36 i think we can likely borrow a lot from the process the openstack vmt uses: 22:20:38 #link https://security.openstack.org/vmt-process.html OpenStack Vulnerability Management Process 22:21:09 a number of other communities outside openstack have adopted some or most of that since we started maintaining documentation about how it's done 22:21:12 fungi: does storyboard support security bugs yet? 22:21:16 it does! 22:21:26 though there's a bit of a wrinkle which needs some work 22:21:39 maybe we can jump straight to that and skip the "private email" option? 22:21:52 the biggest missing piece at this point, i think, is setting up some sort of bridge from configuration management to populating teams through the sb api 22:22:14 fungi: to restrict who can see them? 22:22:21 right now at least, sb only allows admins to configure (and even view the membership of) its teams 22:22:57 so it seems like if we want to start supporting arbitrary teams for adding to private stories, having a way to handle team creation and population through code review would be nice 22:23:20 rather than putting team management on the shoulders of a handful of sb site admins 22:24:21 the next biggest missing bit is that there's currently no feature to be able to set initial teams to subscribe to a new private story, so that would at least for now rely on us providing some instructions to reporters on how to pick an appropriate team to subscribe 22:24:55 that could lead to "missing" bugs which wouldn't be great 22:24:59 e.g., "make sure to add the 'zuul-coresec' team to the story by clicking the ..." 22:25:06 i'm not too worried about that -- when you click the "private" button, it's pretty obvious you need to add a team 22:25:10 or person 22:25:13 ah 22:25:16 right 22:25:26 the ui tries to make it hard for you to miss, but... 22:25:38 so anyway, those are the extent of the rough edges i'm aware of for doing this with sb 22:25:56 in other words, if we've succeeded in getting a person to check the private box, adding the team seems easy. the big challenge is getting folks to that point. :) 22:26:05 that said, we don't currently have anyone doing it (i only just added a vmt team last week and it's apaprently team id #1) 22:26:56 so, um, i'd volunteer to create a team for us, so we can help exercise this. is that unfair? 22:26:59 corvus: oh, so insofar as convincing them to click teh private box, we _can_ also use custom reporting urls which default to private and hide the checkbox for that during story creation if we wanted to have a "go here to report suspected vulnerabilities..." instruction in docs 22:27:33 fungi: that sounds good, though "hide the checkbox" worries me unless there's some other confirmation that "yes this bug is private" 22:27:35 support for that just merged in storyboard-webclient in the past month 22:28:49 i'll have to pull up details on how that bit works 22:29:00 fungi: can a private bug be made public later? 22:29:44 yep 22:29:53 this just hides the checkbox during initial story creation 22:30:11 ah, i see the edit option on existing stories, cool 22:30:43 something like /#!/story/new?title=my%20vulnerability&force_private=true&description=This%20is%20really%20bad... 22:30:58 there are a few settable options through custom reporting urls at the moment 22:31:04 clarkb: as infra ptl, i'll ask you to decide whether i should go ahead and create a zuul-security team, or whether we need to block on automating that... 22:31:23 corvus: considering its largely a one time thing its probably fine to just go ahead and make one 22:31:28 i wouldn't block on automating it for zuul to be able to use it 22:31:47 #action corvus create zuul-security team in storyboard 22:31:56 I assume group membership is relatively easy to change and update later on similar to gerrit 22:32:00 so not too worried about that 22:32:32 more that the openstack vmt is going to need lots of per-project coresec teams in sb when we start moving a lot of openstack projects there, and for _that_ purpose i think openstack-infra could stand to have some sb team management functionality which doesn't require teams to go as a sb admin every time they need a team adjusted 22:32:54 er, to go ask 22:33:25 okay, that gets us reporting, and possibly review for some smaller patches. i would not have wanted to do patch development in storyboard for the recent patch though -- it was large. my guess is we'll probably switch to email for that. but we can coordinate that in storyboard. this gets us a bootstrapping process at least. 22:33:35 one other thing custom reporting urls don't do yet, i think, is allow to prepopulate a project for the initial task 22:33:51 fungi: that would be handy. 22:34:01 i agree 22:34:06 it's been suggested 22:34:52 and yeah, i honestly haven't tried pasting diffs into sb comments yet to figure out how that would work... my guess is not well but i don't have evidence to back that up yet 22:35:42 skipping down the list a bit -- let's talk about cves... i'm imagining that at some point in the future, we may want to request them. but we can probably hold off on that at least until we've actually make a release...? 22:35:54 you can use markdown to quote it but ya may not end up being great 22:35:58 i recommend holding off until you have a release, tes 22:36:00 er, yes 22:36:11 ie, can our policy be "don't request cve's for unreleased vulnerabilities"? 22:36:18 generally mitre discourages issuing a cve for vulnerabilities only present in unreleased code 22:36:35 so maybe even if we introduce a vuln in the future, as long as we don't release it, skip the cve, just fix it (and maybe let folks know on the mailing list) 22:36:44 +1 for that policy 22:36:53 and that is the openstack vmt's policy as well, at least (you'll see in the process document i linked earlier it's one of the classes in our report taxonomy) 22:37:02 cool 22:37:34 we've found having a clear taxonomy of report classifications is an easy way to be able to explain why something doesn't get a cve and/or advisory 22:37:43 now, hypothetically, if we had released last week, and we were working on the current patch checking issue -- should we request a cve for that? it's not exploitable with bubblewrap in place... 22:38:21 I think you would and then you'd just set the severity appropriately. Probably low/medium in this case? 22:39:00 the openstack vmt has requested cve assignment for and sent advisories about vulnerabilities in non-default, non-recommended configurations as long as they're not clearly marked as experimental/debugging or otherwise not for production use 22:40:09 here's what we say on the subject: https://docs.openstack.org/infra/zuul/admin/components.html#attr-executor.execution_wrapper 22:40:17 for example, if your installation and/or configuration documentation clearly stated that the option to disable bubblewrap was insecure and not intended for production usage, we wouldn't have recommended an advisory and cve 22:40:51 and yeah, that documentation in the admin guide seems like exactly that 22:40:55 I guess in that case it does say nullwrap is unsafe 22:41:04 hwoever we also know some users like tobias have to operate in this manner 22:41:20 clarkb: tobiash uses bwrap 22:41:29 at the very least we probably want to communicate the situation even if not getting a cve so that users in that position don't have trouble going forward 22:41:39 corvus: oh right he had to deploy his own openshift to get it working 22:41:41 he has an openshift with admin privs so he can use privileged containers 22:41:46 ya 22:41:52 right, it's never cut-and-dried, so if you know these are risks some users need to take, you adjust your decision based on the information you have 22:41:52 yah ;) 22:42:41 i think our conversation earlier suggested we don't know if any users currently required to task those risks... 22:42:41 that's also a takeaway for the project's documentation authors if they're marking a feature unsafe yet acknowledge that some people need it... this suggests a problem in the overall design ;) 22:43:11 someone was running on vanilla openshift as a service, maybe rcarrillocruz ? 22:44:04 i just can't personally stand behind not using bwrap 22:44:11 yes, rcarrillocruz ran/runs a non-production test env on openshift online 22:44:23 so anyway, for this case, if it had been present in 3.x after official release i'd have suggested not bothering with a formal advisory process unless you knew you had users relying on this clearly documented as unsafe feature, unless you just felt like it was warranted anyway 22:45:24 in particular, you could also consider that as a deciding factor for whether you need to bother with developing and reviewing the fix secretly under an embargo (if the project even wants to support embargoed vulnerabilities) 22:46:38 good point 22:47:11 kurt seifried had a good article posted to the red hat security blog about the trade-offs with embargoed vulnerabilities in open software 22:47:27 the permalinks i had for that are all screwed up so i'm trying to find it now 22:48:07 aha, here we go... 22:48:23 #link https://access.redhat.com/blogs/766093/posts/1976653 The hidden costs of embargoes 22:49:59 yeah, i'm not sure what the right answer will be for us there... 22:51:14 it's a perfectly valid stance to just follow a full-disclosure model with no coordinated disclosure and embargoes 22:51:44 just disclose it as soon as a patch is ready? 22:51:53 depends on what you feel the predominant risk profile is and the ways in which downstream stakeholders consume the software 22:52:19 if we take this as an example embargos probably do make sense so that you can update right away and don't have to turn zuul off for some unknown period of time 22:52:27 (assuming the default setup wasn't secure) 22:52:58 thankfully zuul makes it relatively easy to restart services 22:53:57 yeah, if you feel some vulnerabilities do warrant an embargo and pre-disclosure to some registered subset of stakeholders then i recommend at least having a clear set of rules on how you determine which sorts of reports actually need that level of paranoia 22:54:33 the worst thing is probably 'exposure of secrets' 22:54:34 so that the people handling them don't unnecessarily impose a costly development overhead on fixes for lower-risk vulnerabilities 22:55:17 that's the thing where if there were an exploit for that in the wild, we'd stop zuul and wait for a patch. 22:55:27 makes sense 22:56:30 but yes, if there are reports where you can make a good case for a public development process then you get the usual benefits of people outside your closed set being able to spot potential issues with your proposed solution, the ability to rely on a public ci system to thoroughly pre-test the fix, and so on 22:56:46 fungi: do you have time to start on some zuul-specific docs for this process? 22:57:09 corvus: i can make that a priority for this week, sure. where would you like them to live/be published? 22:57:36 just so i know to ptopose the prose to the right repo 22:57:37 should we put it in the zuul documentation? maybe in the developer section? 22:57:45 wfm 22:57:51 (then maybe on the website, we can make a top-level link for it and deep-link to it) 22:57:59 do we have a specific section on bug reporting? 22:58:01 like "report a bug" "report a security bug" links or something. 22:58:19 fungi: heh, i don't think there's anything more than what's in the contributing file 22:58:52 to compare to openstack's process, we actually have two pages one of which is the instructions on reporting a security bug and the other is the process for the people who will be handling the report 22:59:18 want to keep the instructions on reporting vulnerabilities short and sweet but also flexible 22:59:32 we could put the 'report' section inside the administrators guide even... 22:59:39 so we further include e-mail addresses and openpgp keys reporters can use to securely report via e-mail if they prefer 23:00:23 yeah, you want the bits on reporting to be as easy to find as possible 23:00:33 #action fungi start documenting vulnerability process 23:00:36 referencing them from the admin guide at a minimum would be a good idea 23:00:54 okay, we're at time now... 23:01:00 thanks everyone! 23:01:11 #endmeeting