Allow AnalysisTemplate Jobs to run on the destination cluster that the Stage promotes to.

[kahirokunn]

When Kargo promotes Freight through Stages, each Stage knows which cluster it targets. However, this destination cluster context is not available to the verification (AnalysisRun) phase. AnalysisRun Jobs always run on the control plane cluster (or a single globally-configured cluster via ARGO_ROLLOUTS_ANALYSIS_JOB_KUBECONFIG).

Kargo should provide a way to propagate the Stage's destination cluster information to the verification phase, so that Jobs (and the resources they create) land on the correct cluster.

Related issues:

• Verification in sharded environments should use Argo Rollouts in shard cluster #4189
• Support multiple invocations of same AnalysisTemplate with different args #5062
• Targeted namespaces for AnalysisRun Jobs #4559
• Provide a way from within an AnalysisTemplate Job pod to know which promotion / freight is being tested. #5520

Motivation

E2e verification often requires creating and interacting with Kubernetes resources on the actual destination cluster.

For example, verifying that an AWS Load Balancer Controller promotion was successful requires:

1. Creating an Ingress resource on the destination cluster
2. Verifying that an ALB is actually provisioned
3. Running health checks against the provisioned endpoint
4. Cleaning up test resources

Today, there is no straightforward way to do this because the Job has no access to the destination cluster context. Users would need to manually inject kubeconfig credentials, which defeats the purpose of Kargo already knowing the promotion target.

[krancour]

This is possible already and a duplicate of #5907, where this was already discussed in detail. #5914 is the not yet addressed action item that came out of that thread.

[kahirokunn]

I believe this was closed based on a misunderstanding. The existing solutions (#5907, #4189) cover the single-edge shard case, where one Kargo shard controller maps to one destination cluster. In that topology, controllerInstanceID + ARGO_ROLLOUTS_ANALYSIS_JOB_KUBECONFIG=in-cluster does work.

However, this issue is about a different scenario: a single Stage that promotes to multiple destination clusters. In that case, controller.rollouts.controllerInstanceID only accepts a single string value, so there is no way to route verification Jobs to the correct destination cluster per-promotion.

For example, if a Stage deploys to both cluster-a and cluster-b, the verification Job needs to run on the cluster that was actually promoted to (e.g., to create an Ingress and verify an ALB was provisioned). Today, there is no mechanism to dynamically select the target cluster for the Job based on the promotion's destination.

This is distinct from:

• Support Rollout Edge runs #5907 / Verification in sharded environments should use Argo Rollouts in shard cluster #4189 — single shard ↔ single destination, already solvable
• Support multiple invocations of same AnalysisTemplate with different args #5062 — multiple invocations of the same AnalysisTemplate with different args (related but focused on args, not Job execution target)

Could this be reconsidered?

[krancour]

I will re-open this for further discussion. There was no need for a second issue.

[krancour]

A few excerpts from your previous comment reveal the source of the misunderstanding.

where one Kargo shard controller maps to one destination cluster.

a single Stage that promotes to multiple destination clusters.

For example, if a Stage deploys to both cluster-a and cluster-b,

I know it must seem otherwise, but believe it or not, Kargo has no actual awareness of where things will be deployed, because Kargo doesn't deploy; it only promotes. You've used these terms interchangeably in your issue, but we're very careful in our docs to maintain a distinction between the two concepts. Promotions are mostly git surgery. Deployment is syncing a cluster to what's in git and it's Argo CD's job. Kargo can signal Argo CD to proceed with doing so, but Kargo's very hands-off with the deployment.

What you're requesting isn't possible without a major architectural pivot that would involve Kargo assuming many of the responsibilities that are currently Argo CD's. This would undermine many other key architectural decisions, and isn't on the table. The good news, however, is it should be entirely possible to work around this limitation with careful pipeline design.