Merge pull request #918 from youngnick/gep-917-conformance-principles

Add GEP-917 to document conformance principles

Author: k8s-ci-robot

site-src/geps/gep-917.md

@@ -0,0 +1,166 @@
+# GEP-917: Gateway API conformance principles
+
+* Issue: [#917](https://github.com/kubernetes-sigs/gateway-api/issues/917)
+* Status: Provisional
+
+## TLDR
+
+This GEP outlines the reasons for and principles by which the Gateway API will
+design its conformance and testing regime.
+
+## Goals
+
+- Record why we are doing conformance and what we hope to achieve with it
+- Record the success criteria for the conformance process and associated artifacts
+
+## Non-Goals
+
+- Designing the conformance process at anything more than a very basic level
+- Designing the conformance testing framework or implementation
+- Designing the process for implementations to prove they are conformant
+
+## Introduction
+
+### What is Conformance
+
+Conformance is the creation of a process that allows everyone, implementors and
+users alike, to check that an implementation conforms to the defined spec.
+
+For core Kubernetes, this also allows for the use of specific badges and branding.
+
+It usually includes some form of test harness that can produce a standard output,
+which can be submitted somewhere for validation. The place where the validations
+are held is then the canonical source of information about what implementations
+are conformant.
+
+### Why do it for Gateway API?
+The Gateway API is a large, complex API with many use cases and implementations.
+Not all implementations support the same features, and some features have
+different required support levels.
+
+One of the primary goals of the Gateway API project is to make it safer and
+easier for end users to move their traffic configuration between implementations.
+Without some form of conformance to guarantee the same behavior between
+implementations, this is simply not achievable.
+
+Possibly a better way to say this is that we are looking to have Route resources
+portable between Gateways with a minimum of spec change.
+
+By creating a standard set of conformance tests and information, we can:
+
+- Make it easier for API consumers to understand what a particular API does
+- Make it possible for tooling to be constructed to check for portability
+between implementations
+
+Additionally, as the first project to develop an "official" set of CRDs, we have
+a responsibility to the community to build out a set of best practices for
+similar efforts in the future. Ensuring that whatever we build is reusable for
+other projects will help to lift everyone who works with CRDs.
+
+### What do we need out of conformance for Gateway API?
+Must have:
+
+- Support for only implementing some of the Gateway API CRD resources. Some resources,
+like Gateway itself, are required for all implementations, but implementations
+may choose what Route resources they support. There will probably be some common
+sets of Route resources supported across similar implementations, but this proposal
+expects that we will make calls about what to call that common experience at a
+later date.
+- Support for fields or features that have Core, Extended, and
+ImplementationSpecific support. In particular, it must be possible for
+implementations to only support some subset of Extended fields, and to be able
+to use the framework for their own ImplementationSpecific features if required.
+- A testing suite that can validate that an implementation meets the conformance
+profiles it claims.
+- A way to retrieve conformance information in a machine-parseable way.
+
+## Proposal
+
+### Conformance profiles principles
+
+#### Basics
+
+The Gateway API project defines conformance purely in terms of what resources an
+implementation supports.
+To support a resource, an implementation must support all "Core" functionality
+defined by the resource. Support for "Extended" functionality will be indicated
+separately.
+
+All implementations must support all the Core functions on the following resources:
+
+- GatewayClass
+- Gateway
+- ReferencePolicy
+
+The following resources are optional to support, but have defined behavior if you
+do:
+
+- HTTPRoute
+- TLSRoute
+- TCPRoute
+- UDPRoute
+
+For all of these resources, we should aim to have the usual range of tests for
+both the happy and unhappy paths for various types of operations.
+
+The conformance is versioned - it tracks the required features for a specific
+version of the API, and must be included in and updated by a version bump in the
+bundle version of the Gateway API. (The _bundle version_ is the thing that we
+mark as a "release", that looks like `v0.4.0`, not `v1alpha2`). 
+
+This will enable an implementation to say that it supports a specific version
+of the Gateway API. This is again similar to upstream in that implementations need
+to submit conformance test results for each version of Kubernetes they support.
+
+Because the support is defined in terms of the resources that an implementation
+supports, the conformance is composable, and orthogonal for each object type.
+For example, it's valid to only support HTTPRoute and not TCPRoute, or TLSRoute
+and not HTTPRoute.
+
+#### Interaction with existing support levels
+
+Conformance definitions will ensure that an implementation can provide all the
+features currently marked as "Core" support in the API documentation.
+
+Fields marked "Extended" support will eventually have conformance tests that
+lock in the behavior of that feature, and there will be a mechanism for implementations
+to tell the testing framework what extended fields they support.
+
+#### Testing framework
+
+The Gateway API project will provide a set of tests and harness to run them, such
+that an implementation may point the test harness at a GatewayClass or individual
+Gateway managed by that implementation and have the testing framework deliver a
+report on if it meets the conformance standard. The report must be
+machine-parsable.
+
+There is a _lot_ of work to prepare this framework and introduce the initial
+round of tests, let alone to have complete test coverage. Having a minimal set
+of tests is a requirment for the API to graduate to `v1beta1` API stability level.
+(As per our upstream KEP). It's acceptable to begin with a small set and expand
+outward while the project is in beta, but having a full set of conformance tests
+that cover most of the API scope should be a requirement for declaring the API
+stable.
+
+#### Certification process
+
+The Gateway API project will provide a process by which an implementation may
+submit the results of a run of the conformance test suite to a centralized,
+open repository, and once verified, these results will be used to maintain a
+canonical list of certified conformant implementations.
+
+Ideally, this process should be handled using similar methods to upstream
+Kubernetes, while also learning what we can from what the upstream conformance
+efforts wish they could improve.
+
+
+## Alternatives
+
+There's no real alternative to having some form of conformance testing.
+
+
+## References
+
+[Gateway API Conformance Ideas](https://docs.google.com/document/d/18iECeKMp1OewSGISskv6Chfmjo9u2U0_iUH0jhPdKOk/edit#)
+[Gateway API Conformance Requirements](https://docs.google.com/document/d/1QL-MpIVzqxe32Y2BZ_dYOB8zNsF9c4pnKEIB9ZLt118/edit)
+