Skip to content

Add configurable HTTP timeout for artifact fetching - #1464

Merged
matheuscscp merged 1 commit into
fluxcd:mainfrom
vocarista:vocarista/http-timeout
Apr 21, 2026
Merged

Add configurable HTTP timeout for artifact fetching#1464
matheuscscp merged 1 commit into
fluxcd:mainfrom
vocarista:vocarista/http-timeout

Conversation

@vocarista

Copy link
Copy Markdown
Contributor

Summary

This PR introduces a configurable timeout for HTTP requests used in artifact fetching to prevent indefinite blocking when the upstream source becomes unresponsive.

Problem

Currently, the retryable HTTP client does not set a timeout, allowing requests to block indefinitely if the underlying TCP connection remains open without a response.

Since retries are only triggered after a request fails, indefinitely blocking requests prevent retries from being invoked, leading to worker starvation and stalled reconciliations.

Root Cause

NewRetryableHTTPClient configures retry behavior but does not set HTTPClient.Timeout, leaving request duration unbounded.

Solution

This PR:

  • Introduces a configurable HTTP timeout
  • Sets HTTPClient.Timeout on the retryable client
  • Adds a CLI flag (--http-timeout) with a default of 30s

Impact

This ensures that stalled HTTP requests eventually fail, allowing retries and reconciliation loops to continue instead of blocking indefinitely.

Notes

  • Timeout applies per request attempt; total retry duration is still governed by retry configuration
  • Users may need to tune timeout values based on network conditions and artifact size

Fixes #1463

Signed-off-by: Kumar Piyush <kr.piyush888@gmail.com>
Assisted-by: Cascade/SWE-1.6
@stefanprodan stefanprodan added the enhancement New feature or request label Apr 20, 2026

@stefanprodan stefanprodan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @vocarista

@stefanprodan

Copy link
Copy Markdown
Member

Can you please open a PR in the website repo and add the new flag here: https://github.com/fluxcd/website/blob/main/content/en/flux/components/helm/options.md

@vocarista

Copy link
Copy Markdown
Contributor Author

Sure, I'll do that.

@vocarista

Copy link
Copy Markdown
Contributor Author

I've opened a PR that adds this flag on fluxcd/website:
fluxcd/website#2527

@matheuscscp matheuscscp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@matheuscscp
matheuscscp merged commit 738a3d5 into fluxcd:main Apr 21, 2026
6 checks passed
@vocarista

Copy link
Copy Markdown
Contributor Author

Hi @stefanprodan, @matheuscscp , Please consider backporting this PR to 1.5.x.
We run automated resiliency test suites against live clusters with SLA-bound completion windows. When source-controller becomes unavailable mid-reconciliation, the current behavior blocks helm-controller workers indefinitely (~15 min observed), stalling all subsequent test cases in the queue and causing cascading SLA violations.
With this change and --http-timeout=30s (default), the worst-case block reduces to ~4.5 min (30s × retries), after which the reconciliation fails fast and cluster self-healing can proceed. This makes failure modes predictable and bounded, which is a requirement for our resiliency testing infrastructure.
Given that this is a bug fix with a safe default and no breaking surface, we'd appreciate consideration for v1.5.5.

@matheuscscp matheuscscp added the backport:release/v1.5.x To be backported to release/v1.5.x label May 19, 2026
@fluxcdbot

Copy link
Copy Markdown
Member

Successfully created backport PR for release/v1.5.x:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:release/v1.5.x To be backported to release/v1.5.x enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Helm Controller hangs for ~15 minutes when source-controller goes down mid-reconciliation

4 participants