16:00:06 #startmeeting defcore 16:00:07 Meeting started Wed Dec 16 16:00:06 2015 UTC and is due to finish in 60 minutes. The chair is eglute. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:12 The meeting name has been set to 'defcore' 16:00:39 Hello everyone, raise your hand if you are here for defcore meeting 16:00:57 O/, 16:00:59 #topic agenda 16:01:02 o/ 16:01:30 \o 16:01:39 o/ 16:01:44 \o/ 16:01:49 markvoelker and hogepodge will be either lurking or attending later due to conflicts 16:01:59 * markvoelker lurks lurkily 16:02:27 this will be our last meeting of the year! We will resume on January 6th 16:03:02 o/ (dumb network) 16:03:05 o/ 16:03:12 #chair zehicle 16:03:13 Current chairs: eglute zehicle 16:03:30 #topic TC resolution 16:03:32 o/ 16:03:43 #link https://review.openstack.org/#/c/256438/ 16:04:53 TC has discussed the issue we had been talking about for the last month (but it does feel like a lot longer) 16:05:02 and they have a very strong opinion 16:06:03 They say Linux, but which Linux? :-) 16:06:06 * purp reads latest draft 16:06:12 It *is* getting a little bit more nuanced, though 16:07:03 so, does the TC plan make matching tests to validate these positions? 16:07:17 zehicle great question! 16:07:20 calling mordred 16:07:43 i think right now we do not even have image upload 16:08:11 eglute: There are definitely image upload tests 16:08:25 * markvoelker de-lurks 16:08:29 dwalleck i mean in the current guideline 16:08:47 And there is an existing "upload and boot test", but it may not be specific enough for the task 16:08:54 eglute: Ahh, my mistake 16:08:59 but it will be advisory in the next one i believe 16:09:08 So I think the position of at least some of the TC is that existing tests already actually provide for these. 16:09:45 E.g. the tests in the flag request only run on a linux userland, there are image upload tests as advisory in 2016.01, etc. 16:10:17 Anything using the current remote_client library to run the tests would implicitly require the target VMs to be lilnux 16:10:18 There is probably room for more specificity via more tests, but what's there might be a reasonable "MVP". 16:10:46 markvoelker: There are upload tests in the spec, but I don't think there is a targeted upload and boot 16:10:59 mfisher_ora correct, TC in their resolution are requiring Linux kernel 16:11:08 mfisher_ora: And not just Linux, but specific distros 16:11:20 I saw, I'm just saying that there's already some mechanisms in place 16:11:30 dwalleck: I could be wrong, but I think I recall a test or two that combine upload and boot together into one test. 16:11:30 I'm very certain I can find a distro that does not work with the existing remote client implementation 16:11:40 Whether its a problem or not, but other distros (and even solaris) can pass SOME of those tests (eg: hostname is the same on linux, solaris, etc) 16:11:46 That's why I worry about saying Linux generically 16:12:45 I'd suggest that folks take the "specific distro" question to the TC review if they're concerned about it. 16:12:47 dwalleck and mfisher_ora i think you can also comment on their resolution in gerritt, so that your comments do not get lost 16:13:22 Tje TC discussion seemed to indicate it would be fine if the vendor could boot linux and then restricted the users by turning off the option 16:14:09 that would not be really interop then, since what would the point be of having it, saying you have it, and then turning it off? 16:14:19 So, if you look at the discussion yesterday, it would be worth asking those questions on the review itself. 16:14:49 We also presented this issue to the board a couple weeks ago. This was before the TC resolution. 16:14:50 * purp agrees. 16:14:56 Because cloud vendors can limit access to what they want to sell. Distros, though would have to have the ability 16:14:57 (with rockyg) 16:15:09 eglute: I think the key point there is that the product that is getting an OpenStack Powered mark must at least give customers an option to turn on image upload 16:15:22 eglute: heya! 16:15:31 If an individual deployment of that product chooses to turn that off, fine. The individual deployment isn't seeking a mark. 16:15:38 the simple solution is to offer an OpenStack implementation that has both capabilities. that would pass test and offer specialized function. that's always been OK 16:15:39 mordred we are talking about TC resolution 16:15:41 (see comments on the patch for more discussion of that) 16:15:43 woot 16:16:22 markvoelker, not true. Most public clouds are tested and right now, I think to get a new public cloud listeed in Market, it needs to pass the tests 16:16:29 rockyg: I disagree with your statement above 16:16:33 ultimately, either we need tests that implement the TC solution _or_ we need to change the process 16:16:47 we have also proposed a solution of listing in the marketplace which OSs particular OpenStack product supports 16:16:49 I do NOT think that a vendor disabling the ability to boot linux would be in the spirit of the requirement 16:17:27 #link https://etherpad.openstack.org/p/DefCoreRing.6 agenda (better late than never) 16:17:29 rockyg: in the case of public cluods, the public cloud IS the product though. In the case of a private cloud, those are usually deployments of a product. 16:17:33 openstack provides machines to users - there is no good reason that a user sohuld not be able to run things on that machine 16:18:16 for a private cloud _deployment_ the admin of that cloud can do whatever - but the admin of that cloud also does not need to pass defcore in any way 16:19:02 for a public cloud, if it wants OpenStack Powered, the resolutoin is asserting that the cloud needs to have image upload and that the cloud needs to not arbitrarily limit the os's the user can upload and run 16:19:06 mfisher_ora for Oracle cloud, are you looking for private cloud solutions or as-a-service? 16:19:12 mordred: according to what was said 3 or 4 meetings ago, yes they would 16:19:14 so that if a user wants to upload a HaikuOS image, they can 16:19:15 mordred, true. this is what the vendor offers. that's why the suggestion to handle it w/ the Foundation marketplace listing 16:19:25 since 'anyone should be able to run defcore against any cloud' 16:19:44 mfisher_ora, yes. we have that to hold vendors accountable 16:19:54 * purp chuckles at the HaikuOS reference. 16:19:58 So, question. Like some states that don't allow alcohol sales, if you join a "club", you can buy alcohol there. Could someone put up a private cloud and allow people to join the club that is windows powered by OpenStack? 16:20:01 eglute: our concern isn't for an Oracle cloud at the moment, its for providing OpenStack in Solaris 11 and 12 16:20:04 right. they can run it. but if it does not allow uploading arbitrary os images to the cloud that the user can run, the TC resolution is asserting that they should not pass it 16:20:06 mfisher_ora, is the user made the knowing choice to disable a feature, then it should be OK w/ them 16:20:15 so people can set up clouds using openstack using Solaris 16:21:25 this, btw, is the potential TC position - there are obviously other inputs that defcore must consider - but it's becomming quite clear in the TC that a cloud that does not at least offer VMs is not a thing we'd like to have the name openstack on 16:21:36 if that cloud runs on solaris, or if it runs images that have solaris in them - that's awesome 16:21:40 ok, in this case, user can setup their own cloud using solaris, but unless they want the logo, they do not need to certify it. 16:21:55 certification is what I was told to go after :) 16:24:34 mfisher_ora your product would still be able to pass the certification though, no? 16:24:39 by requiring the ability to allow users to upload arbitrary images, what mechanisms are in place to inspect images for devious things or significant security issues (or prevent them from being uploaded)? 16:24:41 In its current state, no 16:25:36 leecalcote that is excellent question. 16:25:52 and right now, not a lot is in place. 16:26:01 to ensure a good user experience, does the upload mechanism verify or guarantee an image will run? if not, is requiring a public cloud provider to provide a potentially poor user experience a good thing? 16:26:05 there are three tests we can never pass in the current guidelines because they use CLI commands that Solaris does not support (getting partitions and vcpu count). The solaris commands are different, so those tests fail when the command errors 16:26:09 I would suggest flagging the upload images test for the reasons you listed 16:26:14 ....I did 16:26:15 :) 16:26:31 That's exactly the patch I put in that started this mess 16:26:37 mfisher_ora yes, but not for the reasons leecalcote listed 16:27:09 leecalcote: That question has been discussed a bit as well. So for example, at least one person has suggested that in, say, a public cloud, image upload (and pretty much everything else of consequence) be disabled until the user in question has undergone some sort of "this is a valid user and not a fradulent account created for the purpose of creating botnets" sort of screening to be defined by the provider 16:27:32 right. but only one of the 18 current public clouds restricts that today 16:27:38 scuse me - 2 16:28:14 Things like that don't actually fall afoul of the intent discussed thus far, because presumably every cloud has some level on onboarding and once you're onboard, the capabilties are exposed to you 16:28:17 clouds that _don't_ have it are the bad user experience today. now, that being said - there are TONS of improvements that can be made to glance to address leecalcote's concerns 16:28:25 seems like an opportunity for a new project - image scanning (this applies to container images as well). 16:28:35 I suspect that Huawei's doesn't, either. But it's not listed on the market, because it's still not passing all tests. Don't tell Huawei I said that. 16:28:43 They're working on it. 16:28:47 but yeah - I agree with markvoelker - I don't think the current RAX and HP user vetting runs afoul of this resolution 16:28:58 leecalcote: you should have a look at https://review.openstack.org/#/c/232371/ as well as there was some discussion around this sort of thing there 16:29:46 markvoelker: good deal. will do. 16:29:52 eglute: I do not suggest that mfisher_ora flag the tests because of the linux CLI commands. that's one of the other things the resolution is trying to make clear 16:30:05 it is impossible for openstack to accept a patch that fixes the concern stated 16:30:06 An important thing to remember though: whatever mechanisms eventually get into place around that sort of thing need to be market-accepted before they can be considered for an interoperability standard. 16:30:13 so it is a wontfix 16:30:54 mordred which part is wontfix? 16:30:56 we're trying to make it very clear that, since we do not accept as valid a cloud that is unable to run linux, we will not accept a patch that replaces linux commands to test the inbound interfaces with something else 16:31:09 since it is 100% impossible for us to land the patch that would add that ability 16:31:12 since we cannot tet it 16:31:14 test it 16:31:15 (so, for example, if Glance changes the import process to allow some sort of security scanning before an image becomes available, that capability has to become widely deployed before we'll consider it) 16:31:30 but it is totally possible for any cloud that provides vms to run linux on those vms 16:31:49 so any cloud can pass the tests by uploading a linux image and using it to verify the inbound interfaces 16:32:07 mordred so, what you are saying, tempest should test only linux guests? and ignore all other OSs even if people write tests to test Windows, Solaris, etc? 16:32:12 and since there are no restrictions on who can run a linux image, running a linux image in a vm is not an undue burden to expect someone to do 16:32:22 eglute: it can never test solaris or windows guests 16:32:26 mordred: these tests test nothing except that the instance brought up has certain attributes in the instance itself, by running a CLI command. Why would supplying a different remote client be an issue on that? 16:32:29 because those are not open source 16:32:37 it is not possible to commit non-open source code to openstack repos 16:32:48 mfisher_ora: because we can't test it 16:33:10 so the patch to tempest would be untestable -we would have no way of verifying that it's a valid test 16:33:22 What about BSD? It would have the same issue 16:33:32 everyone not-linux would have the same issue 16:33:40 we could do bsd 16:33:44 we can test bsd in the gate 16:33:56 But, it would fail those tests 16:34:01 if someone made a patch that added support for using a bsd guest - it's testable 16:34:04 it would 16:34:04 * purp notes that we're ~halfway through the meeting time. 16:34:07 mordred: can Tempest use closed-source binaries to execute tests as an alternative? 16:34:12 no 16:34:14 it cannot 16:34:33 because that patch would not be testable/runnable by our infrastructure or our community 16:34:53 leecalcote and mfisher_ora would Oracle open source appropriate solaris bits? 16:34:55 patches to tempest to add support for alternate commands on bsd guests _would_ be testable and runable by our community 16:35:22 I think one of the things mordred is trying to get across here is that it's important that we think of DefCore-required tests not just in the context of running RefStack for certification, but as gate tests that run on every commit 16:35:38 Because we want interoperable features to never get broken 16:35:40 markvoelker: ++ 16:35:41 eglute: extremely doubtful. My alternative question is, if Oracle provided CI, would that be enough to add solaris remote_client 16:35:41 that is an excellent point. 16:36:13 mfisher_ora i am guessing that would not be acceptable either based on what mordred is saying 16:36:13 thirdparty CI? 16:36:24 mfisher_ora: no it honestly wouldn't be 16:36:28 okay 16:36:32 mfisher_ora: MAYBE - it would be up to the QA team if they were willing to take on the burden of carrying those patches in the tree with them only being able to be tested in a 3rd party ci 16:36:49 my gut tells me it would be a hard sell 16:36:59 ah - there's mtreinish :) 16:37:04 my gut doesn't have to make guesses 16:37:08 I haven't seen any third party ci setup reliable enough for the community to feel it can get behind 16:37:14 I know we're working on a CI infrastructure similar to what Peter at MS set up, but that's non trivial at the moment 16:37:21 What about changing the tests to be OS agnostic? So BSD could verify a guest? Or Solaris Or Windows? 16:37:23 mfisher_ora: the one thing we've discussed is making the remote client a stable plugin interface. But that interface is no where near ready for external consumption 16:38:02 that's realisitcally not something we can even start planning for/discussing until next cycle 16:38:02 rockyg: again, the question isn't really whether the test can be altered to be more OS-agnostic, it's whether that can be done AND run in the gate. 16:38:11 mtreinish: I talked with dwalleck about that, I thought it sounded like a good idea and I'd like to get to that when things aren't a madhouse here 16:39:25 mfisher_ora: sure, it's definitely something we can have a real discussion about. But there is pre-req work that needs to be done first 16:39:31 absolutely 16:39:32 and that's still in the spec review stage 16:40:21 mtreinish is that something that Oracle can help you with? 16:41:26 * hogepodge lurking 16:41:34 in any case, we spent 2/3 of our meeting on one topic. 16:41:34 eglute: they can participate via the normal community contribution methods. Review, submissions, ml, irc meetings, etc. 16:41:42 which mfisher_ora has been doing 16:41:54 me flashes hogepodge a gang sign 16:41:54 mtreinish glad to hear it! 16:42:23 So, what happens to those tests, then? 16:42:30 unless there are other strong opinions that have not been voiced yet, I would like to move to the next topic 16:42:54 eglute: ++ 16:43:15 it appears that me that positions have gotten entrenched around this question 16:43:16 * markvoelker is happy to continue conversing on this in #openstack-defcore after too, btw 16:43:17 #topic Flag tests requiring multiple tenants and users 16:43:33 #link https://review.openstack.org/#/c/253138/1 16:44:22 Oh, I never did put comments on that one... 16:44:25 this another issue where tests were not made for defcore, but rather for testing. 16:44:27 * markvoelker puts it on to-do list 16:44:28 I am in agreement with hogepodge's research and proposal 16:44:54 I'll comment shortly 16:44:56 I am unconvinced that the cost is worth the benefit at this stage, though I'd be ok with discussing individual tests. 16:45:37 markvoelker which costs? spending on spinning up clouds or re-writing tests? 16:46:04 eglute, ++ 16:46:07 we do not have a ton of tests in defcore in any case, I would be hesitant to flag these just because of the cost of testing 16:46:17 eglute: striking a very large number of tests and capabilities and thereby reducing the capabilities people can rely on 16:46:45 For starters: 16:46:52 +1 we need to be protective of the tests we have 16:47:37 Could we require of vendors but not individual users? Let users skip them? 16:47:39 The rationale around cost here mainly applies to public clouds. While those are very important, note that 62% of openstack clouds in the last user survey are private clouds which generally do not cause a user to incur similar costs 16:47:42 I am all for protecting. If we can get better tests in addition, that would be great as well 16:48:31 rockyg this is really an issue for a third party that wants to verify someone else's test results 16:48:45 So, while I'm in favor of working towards tests that require less resources, I'm not sure it's worth striking down this many tests/capabilities today. 16:48:49 It may be good to account for tenancy/isolation in Defcore by referencing a set of separate security tests. 16:49:13 if I am testing my own public cloud, I have to pay myself. Hopefully I give myself a really good discount. 16:49:17 markvoelker i agree 16:49:55 I'll also point out that to date I have heard zero complaints from the community about the cost of independently verifying compliance. That doesn't mean such sentiment doesn't exist, but might indicate at least that it's not a huge concern yet. 16:50:04 I'd like for us to think about testability as a criteria in scoring, so we can have a better idea of resource requirements beforehand 16:50:11 We can group the tests together as a subset. Then thirdparties can easily say "this feature not tested" 16:50:20 So again: work toward reducing cost, but not sure it's worth weakening interop requirements to this degree 16:50:32 +1 16:50:40 is the cost of running the tests really a factor? 16:51:02 There's no defcore/refstack requirement for third-party audit when testing a cloud, right? 16:51:06 And what would be our threshold for costs? 16:51:19 leecalcote, right 16:51:23 So far. 16:51:32 my patch is very large, I think that set can be winnowed significantly. Initially it was meant to drive discussion. 16:51:35 Isn't the expectation that clouds will be self-testing? 16:51:36 This was the one where I was hoping that we'd focus on sponsorship via a fee for using the trademark vs. cost-optimizations at this level 16:51:43 if running the tests ensures that my cloud automation keeps working, then that seems like a reasonable investment for users/operators 16:51:45 * purp also didn't comment on the review as he said he would. 16:51:54 rather, that companies will be testing their own clouds and submitting results. 16:52:26 catherineD's work in refstack has a huge opportunity to mitigate some of this costs too, as we can have a subset of the guideline that indivuduals can test 16:52:43 leecalcote yes, right now it is self testing and self reporting. hogepodge wants to be able to test clouds independently 16:53:03 hogepodge, that's what I was trying to say ;-) 16:53:10 if everyone could comment on the patch after the meeting, that would be really helpful. We have only 7 min left for other topics. 16:53:25 #action everyone comment on https://review.openstack.org/#/c/253138/1 16:53:39 #topic Remove "cannot flag all tests in a capability" clause 16:53:46 rockyg: split brain here. ;-) 16:53:49 #link https://review.openstack.org/#/c/234824/ 16:53:58 eglute, that makes sense. hogepodge, is this desire driven by the fact that refstack result submitters can doctor the results before submitting? 16:54:05 This is once again an issue of protecting tests 16:54:51 eglute: ...which I'm fairly unconvinced that clause does in any way. =) 16:55:12 while markvoelker suggests that if this guideline does not have teeth, we should remove it, I argue that it could be used as teeth and we should keep it, at least for now :) 16:55:26 leecalcote, the expectation is that it's so easy and embarasing to catch that it's not worth the risk' 16:55:30 eglute: even though we've violated it more than once ourselves already? 16:55:52 zehicle: gotcha 16:56:03 markvoelker it is for vendors, not for the committee, so i would argue that we have not violated it 16:56:11 ++ 16:56:13 My point here is really just that I don't think DefCore would accept a patch that flags all tests in a capability unless there was a really good reason to do so. 16:56:24 I view this as discouragement for vendors to flag all the things. 16:56:32 eglute, that was the intent 16:56:34 So it seems sort of ridiculous to tell people they can't submit a flag request that might be totally valid. 16:56:48 right, since we would not accept it, we also have a documented reason why we would not accept it 16:56:57 markvoelker, the thinking was to stop VENDORS from doing it. DefCore can do it if needed. 16:57:18 I agree that we are not likely to do it blindly so the protection is less needed that we thought in drafting the process 16:57:22 if it is needed would be more of an exception 16:57:24 zehicle: I'll point out that DefCore has no formal membership requirements and most of it's participants also work for vendors 16:57:30 So that provision seems sort of toothless too 16:57:37 And, it's unlikely DefCore would accept a whole class of tests that break a bunch of vendors 16:57:51 markvoelker, maybe toothless but is it causing harm? 16:57:52 It requires two board members 16:58:13 I don't see the urgency to remove the restriction 16:58:26 I do see the need to expect that capabilities have >1 test hoever 16:58:27 I agree with zehicle, I think we should leave it as is 16:58:34 Vendors are the ones who run real tests ... They may discover something that DefCore/RefStack do not know about .. so they should be able to voice the flag issues 16:58:36 If it's hindering discussion about potentially very valid reasons for flagging tests, or requireing vendors to jump through the extra hoop of convincing us to submit the patch for them? Yes, it is. 16:58:36 +1 on more tests 16:59:24 I want to make the DefCore process more approachable for everyone. Provisions like this feel like they just add red tape. 16:59:29 markvoelker, fundamentally I don't really object to removing it since I agree on the humans having judgement 16:59:29 Interesting. The two required members are both -1 on the proposal. Hmmm. 16:59:46 markvoelker have you heard of someone being stopped by this guideline? 17:00:00 Boy, zehicle, you have a rosy view of humanity;-) 17:00:22 eglute: yes, it was brought up in conversation of one of the earilier reviews that flagged all capabilities for a test...which we ultimately allowed anwyay 17:00:28 I'd have to dig for the specific mention 17:00:47 I trust in humanity, good will, and common sense. I vote for leaving if for now 17:00:52 we are also out of time. 17:01:31 If I could ask everyone to review and comment on other outstanding issues, that would be great holiday gift to defcore 17:01:48 please review other issues as listed here: https://etherpad.openstack.org/p/DefCoreRing.6 17:01:52 * markvoelker reminds folks that there's no meeting next week...happy holidays 17:01:54 egallen_, ++ 17:02:07 sorry eglute 17:02:11 ++ 17:02:18 thanks everyone, and happy holidays! we will meet again on January 6th 17:02:26 happy new year. happy to have 1x1 discussions if people want to reach out 17:02:29 Happy hollydaze, folks! 17:02:35 #endmeeting