The pre-submit CI ci.yaml is extremely helpful for testing a change E2E, but in many cases we don't need to run all of the CI to test every change. Addressing this will speed up our development velocity since we don't have to wait for a very long CI pipeline to pass in order merge in a change.

Here is a summary of some of our options:

Option 1: minimal pre-submit; reverts on failures

@yhtang initially proposed the idea of slimming down our pre-submit CI so that only minimal CI is run for PRs, and then the full ci.yaml can be run once the commit is in main. To handle the case where commits in main can cause failures, there can be added logic that will reverts the PR if ci.yaml fails

Option 2: selectively running tests

We can add another "command" similar to /assistant where members of JAX-Toolbox can enumerate the tests needed. So assuming there's four stages:

base -> jax -> paxml
                     |
                     -> t5x

We can comment on a PR something like:

/ci jax

which would jax build and test steps and any descendant jobs (t5x/pax).

/ci t5x paxml

which would run only t5x and paxml

/ci
/ci base

would run everything.