[feature] Opt-in retries for remote cache HTTP requests

[maastrich]

Is your feature request related to a problem? Please describe.

We run a self-hosted HTTP remote cache (Bazel cache protocol, behind CloudFront) for a fairly large monorepo. Under CI load a small fraction of blob requests fail with transient 503s (CDN capacity ramp-up, S3 SlowDown, that kind of thing). We measured around 0.1-0.2% of requests during bursts.

moon currently gives up on the first error. For uploads that's mostly fine (the manifest is skipped and the next run rebuilds), but for downloads a single failed blob out of hundreds means the task ends up with a partial output set and fails hard with task_runner::missing_outputs. So a 0.1% transient error rate on the cache turns into red CI several times a day, even though a single retry would almost certainly have succeeded.

Example from our logs:

[WARN] moon_cache_storage::storage_backend  Failed to retrieve blobs, will attempt to retrieve remaining from other storage backends  storage="http-remote-cache" hash="43d4d1..." expected_count=265 actual_count=176 errors=["Failed to make HTTP request (503 Service Unavailable)."]
Error: task_runner::missing_outputs

Describe the solution you'd like

An opt-in retry setting on the remote cache client, off by default so current behavior doesn't change:

remote:
  api: http
  cache:
    retries: 2

A short exponential backoff (honoring Retry-After when present) on 429/5xx responses would cover the common transient cases. I'd be happy if it only applied to the HTTP client at first.

Describe alternatives you've considered

• Retrying at the infrastructure level: CloudFront origin-group failover can't be used here because behaviors with PUT in AllowedMethods can't target origin groups, and reads/writes share paths.
• Making the failure softer instead of retrying: when a blob can't be fetched after retries, falling back to actually running the task instead of failing with missing_outputs would also solve it. That might be worth doing regardless of retries, but it felt like a separate discussion.
• Wrapping moon with CI-level job retries, which works but re-runs entire jobs for a single lost 28-byte blob.

Additional context

Tested with moon 2.3.2. verifyIntegrity is enabled on our side; the failures above are plain HTTP errors, not integrity mismatches. If a PR is welcome for this I can give it a try.

[maastrich]

Some more data from our rollout that might be useful for whoever picks this up.

We traced our residual 503s with CloudFront access logs: they all show x-edge-result-type: LimitExceeded, and they only happen under moon's own burst pattern. The AWS docs on load testing CloudFront describe the mechanism: a client that resolves DNS once and sends a large burst over a small set of IPs concentrates the load on "a small subset of CloudFront servers" in one POP, which is exactly what a single moon process does when it fetches hundreds of blobs in parallel through one connection pool.

So besides retries, a second (complementary) mitigation would be to spread the HTTP client's connections across the full set of IPs returned by DNS instead of pinning to one or two — that follows AWS's own guidance for this error class. Retries alone would already fix us though, since a refused request generally succeeds immediately on another connection.

[milesj]

I've actually looked into retries before but tonic (which we use for grpc) doesn't support it. grpc/grpc-rust#733

Edit: I'm dumb, I see this is about HTTP. Let me look into that.

[maastrich]

Thanks @milesj if you'd I can look into it and make a pr

[milesj]

I have an idea for this but will require modifying warpgate.