Disable scenarios for architectures other then amd64

[pmacik]

Signed-off-by: Pavel Macík [email protected]

Changes

Currently, all acceptance tests assume that SBO to be installed in Kubernetes or OpenShift clusters deployed on amd64 machines.

While SBO itself can run on the alternative architectures, some of the acceptance tests scenarios cannot.

This PR:

• Introduces tags to diable scenarios/feature for various alternative architectures (@disable.arch.ppc64le, @disable.arch.s390x, @disable.arch.arm64)
• Marks scenarios that cannot run on respective architectures with those tags.

To disable those scenarios, append the following into EXTRA_BEHAVE_ARGS env variable for that acceptance test run:

EXTRA_BEHAVE_ARGS="... [email protected].<arch> ..."

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

• Docs
  included if any changes are user facing
• Tests
  included if any functionality added or changed. For bugfixes please include tests that can catch regressions
• Follows the commit message standard

[baijum]

@pmacik
Instead of using these kinds of deny-lists. Can we use allow-lists? For example @arch.x86_64.
So, this way no need to change test feature files again when a new unsupported platform comes up.

I am not should how to specify the command-line arguments with allow-list. If that is possible, please consider the above suggestion.

[codecov]

Codecov Report

Merging #1109 (4319b98) into master (9a61734) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1109   +/-   ##
=======================================
  Coverage   58.92%   58.92%           
=======================================
  Files          30       30           
  Lines        1670     1670           
=======================================
  Hits          984      984           
  Misses        560      560           
  Partials      126      126           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a61734...4319b98. Read the comment docs.

[sadlerap]

Do we need to have this for v1.0.1?

[pmacik]

@pmacik Instead of using these kinds of deny-lists. Can we use allow-lists? For example @arch.x86_64. So, this way no need to change test feature files again when a new unsupported platform comes up.

I am not should how to specify the command-line arguments with allow-list. If that is possible, please consider the above suggestion.

@baijum we could use allow-list approach but then we would need to tag each and every one of the feature with the @arch.amd64, @arch.ppc64le, @arch.s309x and @arch.arm64 tags (including any new feature or scenario that would be added) and only those that are not supported for the alternative architectures would lack the respective arch tags.

That way we could set the --tags=@arch.<arch> to run those on the particular architecture such as ppc64le.

But that approach would me much more verbose and would require more maintanence.

The current idea and way of the acceptance tests is that we want to run as many of them as possible (ideall all of them) by default to cover the as many of the scenarios/behavior as possible and only if we need to skip some, we would mark them as @disabled... and exclude it by that tag for that particular run.

We use the same approach for excluding tests for a particular versions of openshift

[pmacik]

Do we need to have this for v1.0.1?

@sadlerap Ideally, yes so we can have a matching set of scenarios and released operator correlated by the tag, and since there is still some time before, I'd say let's include it if approved and merged.

This update can be used by IBM Z and P users to run the tests on those platforms

[baijum]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: baijum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [baijum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment