Implementers want running extended conformance tests to be easy

[kflynn]

If you run this:

go test \
    -p 4 \
    ./conformance \
    -run TestConformance \
    -args \
        --conformance-profiles MESH-HTTP,MESH-GRPC

then no extended conformance tests will be run. You have to either enumerate every extended feature you want to test with --supportedFeatures, or use --all-features (which, if you just want to test the mesh profile, also means --exempt-features=Gateway,ReferenceGrant).

This is silly. We should either default extended tests to on, or have a --extended flag that means "run all the extended tests implied by the features included in the profiles I've selected".

[LiorLieberman]

+1, defaulting to add extended makes sense in my opinion.

[LiorLieberman]

cc: @bexxmodd

[youngnick]

Seems like we could either add --extended or default to having all extended features on for selected profiles, and add --core-only or something to turn that behavior off.

[kflynn]

Thinking it over, I think defaulting to enabling the extended features for most (all?) profiles and having a --core-only makes more sense.

[chris-kaiser-7]

Anyone mind if I pick this up? I'm going to go ahead with "--core-only" if thats good.
/assign chirs-kaiser-7

[k8s-ci-robot]

@chris-kaiser-7: GitHub didn't allow me to assign the following users: chirs-kaiser-7.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

Anyone mind if I pick this up? I'm going to go ahead with "--core-only" if thats good.
/assign chirs-kaiser-7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[chris-kaiser-7]

/assign chirs-kaiser-7

[k8s-ci-robot]

@chris-kaiser-7: GitHub didn't allow me to assign the following users: chirs-kaiser-7.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign chirs-kaiser-7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[chris-kaiser-7]

@kflynn Just to be clear for the solution, when you add --core-only flag with a conformance profile, It should run that profile minus the extended features of the provided profiles. Also If no profile is provided should it do nothing or should it add all the core features? Also if combined with --all-features should all features be run or should just core features. Can you combine --all-features and --core-only to do just core features?

Also can you please assign this to me. I don't have permissions for this type of issue I guess.