feat(retry): add exponential backoff retry for 503 errors#375
Merged
Conversation
7a8765d to
eb1c822
Compare
Fixes #339 When creating multiple resources (~10+) with terraform apply, users experience 503 errors from the Clever Cloud API, requiring multiple retries to complete successfully. This creates "retry fatigue" and poor UX. The Clever Cloud API has capacity limitations when handling concurrent requests. Terraform's default parallelism (10 resources) can overwhelm the API, causing 503 Service Unavailable errors. Implemented automatic retry with exponential backoff for API calls that create resources (apps and addons). This approach: - Retries automatically on 503 errors only - Uses exponential backoff: 1s, 2s, 4s, 8s, 16s (max 30s) - Max 5 attempts before giving up - Logs retry attempts for debugging - Generic implementation that can be extended to other status codes - **retry.go**: Generic retry mechanism with exponential backoff - `WithRetry()`: Wraps any client.Response-returning function - `WithRetryConfig()`: Allows custom retry configuration - `DefaultConfig()`: 5 attempts, exponential backoff (1s→30s max) - **retry_test.go**: Unit tests for backoff calculation and retry logic - **addon.go**: Added `CreateAddonWithRetry()` wrapper - **app.go**: Added `CreateAppWithRetry()` wrapper Replaced all direct calls to: - `tmp.CreateAddon()` → `tmp.CreateAddonWithRetry()` - `tmp.CreateApp()` → `tmp.CreateAppWithRetry()` Affected resources: - All addon types (postgresql, mysql, redis, mongodb, elasticsearch, etc.) - All application types (via application/create.go) - Config providers - ✅ Unit tests for retry logic pass - ✅ Provider builds successfully - ✅ Exponential backoff calculation verified Default retry config: - Max attempts: 5 - Initial delay: 1 second - Max delay: 30 seconds - Multiplier: 2.0 (exponential) Users can still use `-parallelism=1` flag for extreme cases, but this should no longer be necessary for most scenarios.
eb1c822 to
f53efd8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #339
When creating multiple resources (~10+) with terraform apply, users experience 503 errors from the Clever Cloud API, requiring multiple retries to complete successfully. This creates "retry fatigue" and poor UX.
The Clever Cloud API has capacity limitations when handling concurrent requests. Terraform's default parallelism (10 resources) can overwhelm the API, causing 503 Service Unavailable errors.
Implemented automatic retry with exponential backoff for API calls that create resources (apps and addons). This approach:
Retries automatically on 503 errors only
Uses exponential backoff: 1s, 2s, 4s, 8s, 16s (max 30s)
Max 5 attempts before giving up
Logs retry attempts for debugging
Generic implementation that can be extended to other status codes
retry.go: Generic retry mechanism with exponential backoff
WithRetry(): Wraps any client.Response-returning functionWithRetryConfig(): Allows custom retry configurationDefaultConfig(): 5 attempts, exponential backoff (1s→30s max)retry_test.go: Unit tests for backoff calculation and retry logic
addon.go: Added
CreateAddonWithRetry()wrapperapp.go: Added
CreateAppWithRetry()wrapperReplaced all direct calls to:
tmp.CreateAddon()→tmp.CreateAddonWithRetry()tmp.CreateApp()→tmp.CreateAppWithRetry()Affected resources:
All addon types (postgresql, mysql, redis, mongodb, elasticsearch, etc.)
All application types (via application/create.go)
Config providers
✅ Unit tests for retry logic pass
✅ Provider builds successfully
✅ Exponential backoff calculation verified
Default retry config:
Users can still use
-parallelism=1flag for extreme cases, but this should no longer be necessary for most scenarios.