Decoupling API versioning and Feature versioning for features turned on by default

[JeromeJu]

This issue tracks progress on the API/feature versioning decoupling as the existing v1 migration blocker. It aims to initiate the discussion on the impact of this towards stakeholders and reach consensus on the approach moving forward.

Context

Currently, the behavior of enable-api-fields depends on the CRD API version being used. In v1beta1 CRDs, beta features can be enabled by setting enable-api-fields to beta or to stable, but in v1 CRDs, beta features can only be enabled by setting enable-api-fields to beta. The coupling of API versioning and feature versioning has been a blocker to the v1 migration.

As discussed in the API WG, we would want to decouple API versioning and feature versioning for features
that are turned on by default. If a feature flag was enabled by default but not enabled later on, it would
be considered a breaking change. And it could cause confusion for end users when we are moving to future
stable versions with unstable features. An approach needs to be carried out for avoiding such breaking
changes.

Use case

i.e. resolver is turned on by default in v1beta1 but cannot be migrated to v1 directly since it is not a beta feature in the v1 API, which could potentially cause breaking changes for users.

Proposed Options

There are some ways that we could use for moving forward:

• Adopting per feature flag
• Promote features to stable in stable versions also

See decouple API and feature versioning problem statement and adopting per-feature flag proposal for more details.

related: #6579

[vdemeester]

I am not sure I follow entirely 🤔 especially as there is not really such thing as feature versionning. The only "version" we have are the API version and the "code/release" version.

Reading the use case, my understanding is that, at least in the example, you can have something that is "usable" in both API versions, but by default on one (v1beta1) but not on the other (v1) ? If that's the case, then, this goes back to the choice of the enable-api-fields values (alpha, beta and stable) and there "enablement" between different API version.

I do still believe, aside from naming, the current behavior is indeed a bit confusing. The feature flag should be independent from the API version used. A feature is experimental or stable (or in between if we consider alpha->beta->stable flow), no matter what the API version is. Enabling by default the resolvers on v1beta1 but not on v1 is indeed a bit confusing. If we enabled it by default on any API version we publish, this means we feel it's stable enough to be enabled by default, imo.

[dibyom]

Reading the use case, my understanding is that, at least in the example, you can have something that is "usable" in both API versions, but by default on one (v1beta1) but not on the other (v1) ? If that's the case, then, this goes back to the choice of the enable-api-fields values (alpha, beta and stable) and there "enablement" between different API version.

You are correct!

I do still believe, aside from naming, the current behavior is indeed a bit confusing. The feature flag should be independent from the API version used. A feature is experimental or stable (or in between if we consider alpha->beta->stable flow), no matter what the API version is. Enabling by default the resolvers on v1beta1 but not on v1 is indeed a bit confusing. If we enabled it by default on any API version we publish, this means we feel it's stable enough to be enabled by default, imo.

Agreed. For features that are currently in beta, we have a couple of options when we move to v1 as the storage version

1. we do nothing - this would mean features that are on by default today are not on by default anymore - this is poor UX and a breaking change for the user

2. we enable (some or all) beta features on by default - basically we change the first requirement here. Note that the rest of the beta features policy still apply i.e. we can make breaking changes with the proper notice.

3. we beta features such as resolvers to GA first - IMO, we should not rush pushing features to GA just to be conformant with our current feature gates implementation.

To me, 2. seems like the least bad option right now. Open to other ideas and feedback!

/cc @lbernick

[lbernick]

I'm confused what it would mean for resolvers to be both on by default (i.e. on even when enable-api-fields is set to stable) but still be a beta feature. Does this mean it would follow the beta stability policy (can be changed w/ 9months warning) even though it is a "stable" api field?

It seems like to satisfy both requirements we'd need to make the default value of enable-api-fields "beta", but that's not backwards compatible.

[dibyom]

You are right, the naming of the flags makes this confusing. The intention would be to decouple the stability level of the feature from whether it is enabled by default in a particular API version or not.

[dibyom]

An example to illustrate this issue:

In #5938 we switch the reconcilers to use v1 types. Now, this should be not be a breaking change for the user at all but what we see is that some existing v1beta1 pipelineruns are now failing e.g. test: TestYamls/yamls/v1beta1/pipelineruns/pipelinerun-with-final-tasks.yaml is failing validation with

            message: 'Pipeline arendelle-xxmg8/clone-cleanup-workspace can''t be Run; it has
              an invalid spec: must not set the field(s): tasks[0].taskref.params, tasks[0].taskref.resolver' 

(see full logs)

[lbernick]

I think the issue Dibyo is pointing out here is a separate but related issue; I've created #6616 to track this.

[vdemeester]

I'm confused what it would mean for resolvers to be both on by default (i.e. on even when enable-api-fields is set to stable) but still be a beta feature. Does this mean it would follow the beta stability policy (can be changed w/ 9months warning) even though it is a "stable" api field?

Let's assume instead of "stable, beta and alpha", the value for enable-api-fields would be "one,two,three". If we consider resolvers to be behind "two", they would be enabled only when enable-api-fields is set to "two", no matter what in what API — aka you could only use resolvers, in v1beta1 or in v1 if the value is "two", or "one".

The "beta" status of the API is different from the "beta" status of a given feature.

A very good example of how confusing this behavior is :

• Today, I am using v1beta1, and I am using resolvers (let's say the git and bundle resolvers).
• v1 is available, so I want to migrate to v1 (because why not 👼🏼)
• I migrate all my object to v1, still using resolvers.
• I try to apply them, and they get rejected because well, it's not allowed (as resolvers are not enabled by default on v1)

This is so confusing and counter-intuitive for users.

It seems like to satisfy both requirements we'd need to make the default value of enable-api-fields "beta", but that's not backwards compatible.

What wouldn't be backward compatible ? We are in a situation where the only "way forward to be backward compatible" (what a weird sentence 😂) is to mark resolvers as stable, as well as everything that is behind enable-api-fields: beta today before changing the meaning of enable-api-fields...
And I really think this is something we need to do 😅

I also think we need to rethink the enable-api-fields flag and figure out if it's not bringing more issues than having per-feature feature-flags. It doesn't matter if it's more work for us (developer/maintainer of tektoncd/pipeline), it has to be useful, consistent and easy for the end-users.

[dibyom]

I also think we need to rethink the enable-api-fields flag and figure out if it's not bringing more issues than having per-feature feature-flags. It doesn't matter if it's more work for us (developer/maintainer of tektoncd/pipeline), it has to be useful, consistent and easy for the end-users.

+1. @jerop and I were just discussing the Kuberentes style feature flag support where each feature has a flag, and the stability level is tracked in code in a struct (vs via docs like we have today)

@vdemeester making sure I understand what you are saying - are you saying the only way forward is moving resolvers and current beta features to "stable" or can we set enable-api-fields to "beta" once we move the reconciler and storage versions to v1 (to match the current reconciler behavior)?

[vdemeester]

@vdemeester making sure I understand what you are saying - are you saying the only way forward is moving resolvers and current beta features to "stable" or can we set enable-api-fields to "beta" once we move the reconciler and storage versions to v1 (to match the current reconciler behavior)?

Yes, that's what I am saying from "on top of my head" 😝