Monday, 2017-08-07

*** emagana has joined #openstack-tc01:23
*** emagana has quit IRC01:28
*** emagana has joined #openstack-tc03:08
*** emagana has quit IRC03:20
*** david-lyle has quit IRC04:38
openstackgerritArtur Basiak proposed openstack/governance master: Add monasca-events-api
openstackgerritDuong Ha-Quang proposed openstack/governance master: Add kolla-ansible-plugins
*** cdent has joined #openstack-tc09:28
*** sdague has joined #openstack-tc09:47
*** dtantsur|afk is now known as dtantsur09:48
openstackgerritDuong Ha-Quang proposed openstack/governance master: Add kolla-ansible-plugins
openstackgerritWitold Bedyk proposed openstack/governance master: Update goals status for Monasca
*** cdent has quit IRC12:02
*** cdent has joined #openstack-tc12:33
cdentlet the great CD debates begin!12:47
* fungi deploys from trunk... in the gate tens of thousands of times a day?13:12
fungineedless to say, the rate of success there is about what you'd expect ;)13:12
*** gcb has joined #openstack-tc13:28
*** lbragstad has joined #openstack-tc13:36
dhellmannwe should qualify that question by asking for folks who deploy from trunk and expect API stability13:37
dhellmannbecause if we have a lot of trunk deployers who don't care about api stability, that's also an interesting answer13:37
cdentour lack of reliable way to ask questions is problematic13:39
dhellmannI understand the foundation's reluctance to have lots of separate surveys going, but we do need to work something out.13:42
dhellmannin this case, based on past user survey results, adding a question there is likely to produce the result that extremely few (if any) users are deploying from trunk13:43
dhellmannand we have no way of knowing if that information is accurate, or biased based on survey respondants13:43
cdentwe could break some stuff and wait for people to come out of the woodwork13:44
dhellmannyou would expect anyone deploying from trunk to be relatively well engaged in the community, though, so maybe asking on the operators list would give us useful answers?13:44
dhellmanncdent : we could make nova's API for creating a server return a message13:45
dhellmann"please contact the TC about your trunk deployment by emailing ..."13:45
cdentI’m no longer convinced that that expectation holds. That was the main thrust of Josh’s message: if these people exist how come we can’t see them?13:45
cdentI like it!13:46
dhellmannI think we probably have 1-2 very large older deployments still doing CD from upstream. Whether that's trunk, and how far behind, was never really clear13:46
dhellmannRAX is the one usually cited when this discussion has come up in the past13:46
dhellmannspeaking of, we might find some info by reviewing mailing list archives. at least some names of companies to contact directly.13:47
dhellmannI guess it depends on how much time we want to invest into research.13:47
cdentyeah, I’m not sure. To some extent there are enough habits burned in, that changing would be very disruptive.13:48
*** marst has joined #openstack-tc14:09
sdagueI think there is also a discipline in using the CD model to ensure the release to release path is smooth, because if you solve the any to any+ problem, you get the N -> N+1 problem for more or less free14:24
sdaguewhich makes the release sanity checking a lot smoother14:24
dhellmannright, I think it makes sense for us to keep doing the upgrade testing in our CI. I'm just not sure it makes sense to spend lots of time agonizing over API stability between official releases.14:25
sdagueyeh, I tend to think it's just discipline to think through all the API changes as if they were going to be in the release, as it means you don't have to consider it as part of the other things you have to think about at release time14:34
cdentyeah, for the most part, the habits are pretty useful14:40
cdentbut I suppose there might be times when they could be usefully broken14:40
* cdent shrugs and goes home14:40
sdagueyeh, everything needs an exception criteria. It's just norms actually let you think less about each decision. Always do it this way unless there are exceptional circumstances. We already put in the exception explicitly about closing security issues.14:41
smcginnisThe discipline is good, but I still have yet to see where we've ever stated officially that we would support CD.14:44
smcginnisI've seen it stated plenty of times for reasons why we can't just quickly fix a mistake though.14:44
*** cdent has quit IRC14:44
*** gcb_ has joined #openstack-tc14:45
*** gcb has quit IRC14:47
sdaguesmcginnis: yeh, I think it is all part of the culture of the "before times"14:49
sdaguewhere everything wasn't written down14:50
smcginnissdague: Which is fine. (My take too btw). But if it is still something we want to adhere to, I want the outcome to be that we declare that publicly in writing.14:51
*** gcb_ has quit IRC14:52
sdaguesmcginnis: sure, that's fair14:54
smcginnisOr more accurately, I want us to declare that we don't care unless it's an official release, but either way, I want it more explicit. ;)14:57
*** emagana has joined #openstack-tc15:02
*** david-lyle has joined #openstack-tc15:04
sdagueyeh, explicit is fine15:05
*** cdent has joined #openstack-tc15:08
*** emagana has quit IRC16:01
*** emagana has joined #openstack-tc16:01
*** emagana has quit IRC16:05
ttxalso worth noting, we already consider that a vulnerability introduced in master but not released is not worth an OSSA16:07
ttxso we already have different commitments for people tracking master and people tracking releases16:08
smcginnisttx: Good point.16:09
*** Qiming has quit IRC16:18
*** Qiming has joined #openstack-tc16:20
fungialso, i don't every remember openstack officially stating it would "support" deployments from arbitrary points in master branches, or continuous deployment. lots of _individuals_ in the project said they kept the possibility that people might do that in mind, which is cool... i'm all for enabling additional use cases where we can without overtaxing ourselves. but a declaration of support for that seems17:57
fungifar too strong17:57
fungithere's lots of ways people can and do use openstack software, but just because there are people doing things with it doesn't necessarily mean we can officially support all those things17:59
fungithe recent statement on postgresql seems like a relevant example17:59
fungiyou might be able to use openstack with postgresql, and upstream developers aren't actively trying to make that harder for you, but we also can't promise to support it18:00
fungii do recall some teams in the past pushing for continuous deployment support on their own, but openstack as a whole hasn't prioritized that use case that i've ever seen18:01
fungiand the organizations i know of who do that or did it in the past coupled their deployment process with an entire shadow deployment and test infrastructure where they could shake out/postpone breaking changes and integrate their local forks (because they pretty much always have them) before a commit from upstream master ever trickles into production18:04
*** emagana has joined #openstack-tc18:10
*** dtantsur is now known as dtantsur|afk18:12
sdagueyeh, well you have to keep CDing in mind when writing software so that you have continuous transitions for upgrade18:20
smcginnisfungi: ++18:52
sdaguefungi: you clearly need your own local CD testing infrastructure if you are going that way18:59
sdaguebut, if the upstream software isn't even trying to work in a CDing manner, you can't do local CD19:00
fungisure, just saying i think it's so far been possible by accident/conscience, not because we declared officially to provide support for doing that19:01
fungiteams have perhaps been trying to avoid gratuitously breaking it19:02
dimsfungi : so write down what we are doing now already? (tell projects to continue to honor the CD target)19:18
cdentdims: do all projects do that or just some?19:24
smcginnisMaybe we need a follows-cd tag.19:27
dhellmannwe're talking about 2 different aspects of CD, though, right?19:28
dhellmannsdague correctly points out that upgrades are important19:28
smcginnisI think you need to keep D in mind in order to have upgrades, but the C part is another apect.19:28
dhellmannthe thing I'm more worried about is artificially locking us into supporting an API that isn't fully baked19:28
smcginnisdhellmann: ++19:28
dhellmannor where some bug slips through, as in the keystone case19:28
smcginnisAnd DB migrations.19:28
dhellmannI think we can relax the rules on the latter, while continuing to support the former19:29
dhellmanndb migrations would be part of the upgrade testing, right?19:29
dhellmannor are you saying it's OK to change a migration before a release?19:29
sdaguedhellmann: I hope not19:29
smcginnisdhellmann: Yes, but we've had cases where we've realized something wasn't the way we want, so rather than changing the migration to make it right, we've had to then add yet another migration because "someone might have picked that up already".19:30
smcginnisI would like to be able to just fix the intial migration rather than keep around crud just because of a "maybe".19:30
dhellmannthat would make our upstream upgrade testing a bit more complex19:30
dhellmannor would it?19:30
smcginnisOr have it clear that we should fully support that "maybe" and be strict about it.19:31
dhellmanndo we deploy from the last full release, then test upgrading to master?19:31
smcginnisdhellmann: I don't think it would.19:31
smcginnisLast full release in grenade.19:31
dhellmannoh, ok, then it seems less risky19:31
smcginnisThat's the boundary I would prefer to operate on. Or at least last milestone "deliverable".19:31
sdagueright, so here is the thing, if we think we ever want CD to be a thing we do need to bake some of these ideas in that once things are released they are real19:32
dhellmannmilestones seems reasonable, since I could see someone doing test deployments with those19:32
sdagueI do get that's harder engineering and requires more thought19:32
dhellmannsdague : right, but I think we're talking different definitions of "released" (committed vs. tagged)19:32
smcginnissdague: I don't think we have agreement that we all "want CD to be a thing" though.19:32
smcginnisBig C continuous.19:32
dhellmannthere's also a question of how far to extend that, to just deployer facing changes or all the way to end-user facing changes19:33
dhellmannwhich is why I was pointing out the difference we might want between upgrade testing and API testing19:33
sdagueI'm not sure why we think milestone tags are special at this point. We don't have soft freezes for those any more.19:35
smcginnisWhich is why I would rather be between releases really. Especially since that's all we test with grenade.19:35
smcginnisI would much rather support Ocata->master than master~1->master19:36
sdagueI think that if we ever want to close the gap with out of tree features and providers being 2 - 5 release behind upstream, we need to encourage the CD model19:36
sdaguebecause no one is going to ocata -> master if they can't master -> pike19:37
sdaguewhich means no one is going to do anything other than consume stable branches, 3 - 9 months after we publish them19:37
dhellmannI'm not sure how I'm seeing why suggesting that APIs only have to be stable on release boundaries breaks anyone's ability to upgrade.19:38
sdaguewhich puts their deployed experience a full year out of step usually with where the source code is, which makes it quite hard to integrate operators in19:38
sdaguebecause if they deploy at a random git commit, that's the interface their users start writing to19:38
sdagueand then if they upgrade away from it, boom19:38
dhellmannright, well, I think it's a bad idea to encourage that19:38
dhellmannmaybe I just think CD is a bad delivery model for a stack this complex; I don't know19:39
dhellmannI'm looking for ways to introduce some slack into the upstream processes19:39
smcginnisdhellmann: Agree 100%19:39
cdentmaybe the stack being this complex is the real issue? ;)/219:41
smcginniscdent: Well, there is that. :)19:41
dhellmannyes, that's completely fair, too19:41
sdaguepersonally, part of what made me passionate about OpenStack is that it functioned in an already ready to deploy mode19:42
sdagueif we're taking that off the board, that would make me quite sad19:42
cdentthere’s a different between ready to deploy and continuously deployable from master, isn’t there?19:43
sdaguenot if you want to support upgrades19:43
smcginnisI don't agree with that statement.19:43
sdagueif you say you can deploy from any commit, but you might be stuck, that's not really saying you can deploy from any commit and have long term working software19:43
cdentwhich “that” smcginnis ?19:44
smcginnisSometimes it's easier to support upgrades by not locking yourself into a mistake.19:44
sdaguesmcginnis: easier for who?19:44
smcginnisSo I absolutely do not see CD off of master as ar equirement for upgradability.19:44
smcginnissdague: All involved.19:44
dhellmannwell, we wouldn't be saying "you can deploy from any commit"19:44
dhellmannwe would be saying "we work toward release boundaries, pick one"19:45
sdaguesmcginnis: I don't understand how it is easier for end users to consume if API features come and go19:45
smcginnissdague: They don't come and go if they go off of release boundaries.19:45
dhellmannsdague : you're assuming a configuration that is something we would specifically not support19:45
sdagueit's easier for developers of openstack, because they can fix more mistakes19:45
smcginnisBy all means test what's on master and don't wait until release to even look at it.19:45
sdaguedhellmann: I'm saying the grey state right now19:45
smcginnisBut don't put that on your end users.19:45
sdaguesmcginnis: how do you test it without having consumers19:46
dhellmannsdague : well, I'm saying we would not encourage users to deploy from untagged commits *and then* we would change the policy to say API changes are only stable at tagged releases.19:46
sdaguedhellmann: right, I understand what you are proposing. I think we as a community loose a lot in the process19:46
sdagueand I'm trying to state where we currently stand19:46
dhellmannok, well, it sounds like you're backing that with a justification of if they do something other than what we want to support we're somehow doing them a disservice19:47
sdagueI'm saying that if you relax the dev teams striving for a CD able upstream tree, it's not really possible to retrofit it later19:48
sdagueit's like retrofiting pypy support or something19:48
dhellmannI'm not sure if I agree or disagree, but I'm also not sure that's a goal I would have.19:49
cdentsdague: so part of your concern is that if we give it up we can’t get it back?19:50
sdaguecdent: yes19:50
cdenta fair concern19:50
sdagueand, that even if the number of folks doing it is small, that's incredibly valuable feedback to the dev teams, because they can expose issues before we hit stable releases19:51
cdentI’m struggling to develop a solid opinion19:51
sdagueand it would be great to have more folks in that camp because it also gives us better stable releases19:52
cdentI wish we had a stronger more visible set of participants19:52
sdaguecdent: I do as well19:52
sdaguebut we will never get those folks if we walk away from the goal of supporting CDing19:53
sdagueboth RAX and HP in their prime for public cloud were trunk chasing. They both stopped when they effectively decommitted from the project.19:54
sdaguewe went through a lull19:54
sdaguenow we're getting a lot more public cloud folks19:55
sdagueand I hope / expect more of them will get on this bad wagon19:55
sdagueit fits closer to the public cloud model where you've got skilled folks deploying via ansible with a smallish number of custom site fixes. It's not product turnkey stuff.19:58
cdentmaybe the foundation should make a trystack-like thing which is CD’d19:58
cdentfree for 24 hour vms, ripe for abuse!19:59
fungiin the latest case, i think it's more that if you as a cd'er start relying on a feature which hasn't seen a release yet, then you should be mindful the behavior of that feature may still change prior to release and prepare accordingly20:05
cdentfungi: the problem with that model is that a user may not know they are talking to a cloud that is cd’ing20:07
cdent(although I’m not sure what you mean by “the latest case”?)20:07
fungican we mark new interfaces/features "experimental" somehow before they hit an official release?20:07
fungialso "continuously deployable from master" isn't necessarily the same thing as "continuously upgradable from any arbitrary point in master"20:08
smcginnisI think there was some talk of using microversion x.9999 to do that.20:08
cdentsmcginnis: where x.9999 is > ‘latest’?20:08
fungialso, if we have to start supporting features and not breaking regressing them as soon as they hit master, we're going to see a proliferation of "feature branches" just so devs have an out to avoid anyone depending on something which is still in flux20:09
smcginniscdent: Something like that. I just remember hearing something about it in passing. I'm not sure about practical implementation.20:09
sdaguefungi: why?20:09
sdaguefungi: this has been the norm state for the last 7 years20:09
fungiwill we then declare hopping between different feature branches or upgrading back and forth between master and feature branches is supported once cd'ers realize they can't get the bleeding edge features they want from master any longer?20:10
fungii wonder where that road ever ends, really20:10
sdaguefungi: except, we're already on that road20:10
sdagueand we've been for a long time, and we didn't have that20:10
sdaguewhat we got was the invention of microversions so that you could stay on that road and still signal to your users what's going on20:10
smcginnisMaybe we do need to start using feature branches more if we're going to actually declare this as a principle.20:10
cdentdon’t we have experience that says that feature branches often cause more trouble than they are worth20:11
cdenthowever, I think it is important that we don’t let the experience of the bigger more complex projects (e.g. nova) to necessary implicate other less complex projects20:12
fungisdague: i don't see where making a tc resolution that to be an official openstack service you must provide support for anyone who wants to run and upgrade between changes on your master branch is "the norm for the last 7 years"20:13
fungiwe have devs who have tried to make that feasible as much as possible without compromising their ability to advance their platforms20:13
sdaguefungi: nope, it was just the baked in culture that I showed up20:14
*** mriedem has joined #openstack-tc20:14
fungiwhich is more what seems like "the norm for the last 7 years"20:14
dhellmannI think the reality was that some teams adopted that stance more than others20:14
sdaguedhellmann: sure, that's fair20:14
fungisimilarly, as soon as we land methods/functions in an unreleased branch of a library, do we need to do full version bumps if we alter their behavior before the next tag?20:15
dhellmannwe only consume libraries at releases, so we have not enforce that20:15
sdaguefungi: I think that the set of constraints you give yourself for it isn't all that constraining. But that's just personal opinion. It's about sane deprecation, data model migrations, and having a progressively changing API20:15
fungieven if it's to fix an undesirable behavior we didn't catch in review?20:15
sdaguedhellmann: we release libraries more than every 6 months20:15
dhellmannsdague : we could release services more often, too20:16
fungiwell, "we" aren't the only ones consuming the libraries we produce either20:16
dhellmannand we still only have stable branches for libraries every 6 months20:16
fungiso if someone wants to continuously deploy any of our libraries, the same concerns come up as far as i can see20:16
dhellmannwe haven't supported CD of the libraries since we switched oslo off of alpha releases20:17
fungii'm curious how it was "supported" prior to that20:17
sdaguefungi: one key difference is you can pip install the libraries20:17
dhellmannalthough I'm not sure that's written down20:17
sdaguewhich you can't do for the projects20:17
dhellmannyou can pip install some of them20:17
sdaguedhellmann: to a working state?20:18
fungiyou can pip install the services too, you just need to pip install them from a local tarball/path and do some steps afterward to get configuration and data files into the right places, right?20:18
dhellmannsdague : can you install an RPM and get a working nova?20:18
sdaguefungi: ok, I guess I20:18
sdaguedhellmann: you get a lot closer for sure20:19
mriedemrpm lays down your config files and creates nova user/group20:19
fungiyou can't pip install most of the services directly off pypi, but that doesn't mean that you can't use pip to install them at all20:19
sdagueanyway, I think we're sort of in circular camp now20:19
mriedemand rootwrap etc20:19
fungirevisiting the continuous deployment argument, it seems that maybe api behaviors are "special" in this regard because that api has a version number independent of the software release version, and (currently) no way to mark api features under development as experimental20:30
fungiwhereas features in a library have only one version, and the version for an unreleased library is clearly an experimental version number20:31
mriedemthere is actually a status with the versions
mriedemnova doesn't have EXPERIMENTAL anymore but i think glance v3 had/have it20:32
mriedem"id": "v2.6",             "links": [                 {                     "href": "",                     "rel": "self"                 }             ],             "status": "EXPERIMENTAL"20:34
mriedemi don't know how glance moves something out of experimental20:34
sdagueI think the project history with experimental was that it turns out to not be incredibly useful in practice20:34
mriedemi'm not advocating it,20:34
sdaguebecause by the time your consumers deploy things to experiment with them, you've already committed to it or deleted it20:34
mriedemjust saying, there was a system before microversions20:35
sdaguebecause they are releases behind you20:35
sdaguemriedem: yeh, I wasn't saying you were, I was providing context about why it went away20:35
mriedemi hear that you're saying that i wasn't saying something, and am pleased.20:38
* mriedem investment in is already paying for itself20:40
*** mriedem has quit IRC20:56
fungijust wondering how you can have an intermediate state between versions that't clearly unsupported, and with api versioning we don't presently have a mechanism to indicate to end users that they shouldn't rely on a feature (specifically because continuous deployment may be exposing that to them when it's not yet fully baked)21:02
fungian alternative, perhaps, is to guard in-progress api features behind testing-mode switches that we tell operators (via documentation) not to turn on21:04
fungiallowing us to run some jobs with those experimental features enabled while avoiding cd'ers from exposing them to end users before we're ready to support their behaviors21:04
dimsfungi : ++ (k8s uses --feature-gates for example - with clearly marked features and ALPHA/BETA designations)21:20
*** marst has quit IRC22:36
*** sdague has quit IRC23:21
*** cdent has quit IRC23:22

Generated by 2.15.3 by Marius Gedminas - find it at!