What problem are you facing?
I've observed use-cases where provider-terraform resources call out to APIs that do not properly implement rate limiting. The API owners get cranky about the number of retries made by Workspaces orchestrated by this provider since the retry mechanism is based on the Rate limit configuration defined in crossplane-runtime in addition to having no facility, currently, to adjust re-queuing of reconcile requests when there is an error.
What could help solve your problem?
Ideally, crossplane-runtime would support some mechanism to specify RequeueAfter or DoNotRequeue as part of Update/Create/Delete/Observe Result structs. This would be ignored when there is no error present but -- in the case of an error -- would be used to adjust the re-queue behavior of crossplane-runtime reconcile for these actions. I will likely file an issue there to see if this is at all possible, but I know this has been a bit of a challenge in the past. With this, we could expose a new MR field to accept a duration that will be fed to this configuration on error.
More ideally, downstream APIs would implement rate limiting and not care about this, but that is a different story :)
Less ideally, the Workspace reconciliation loop can be changed to conform to existing expected behavior, with an observability trade-off, in an opt-in fashion. I've implemented this pattern before in a native provider with a similarly un-rate-limited downstream API.
- In Update/Create, track
lastSyncFailed timestamp in status.atProvider. We can also track lastSyncFailedCondition to capture the status.condition(s) data present when the failure occurred
- In MR, create new
spec.forProvider.retryAfter field (naming can be adjusted) that takes in a duration string declaring the desired minimum duration before the controller should attempt a full reconciliation of the resource. If null, existing behavior is preserved. We may also opt for an annotation-driven approach, to avoid polluting the schema with something that really should be solved in crossplane-runtime
- In Observe, check
lastSyncFailed timestamp against current time.
- If the duration configuration is provided with the MR and the duration has not yet been met, we will return successful status from Observe, short-circuiting the remainder of the reconciliation loop
- If the duration configuration is NOT provided OR the duration has been met, then we proceed with the full reconcilation loop, as required
The trade-off here is that MRs will report false readiness/sync states during the periods where they are skipping full reconciliation. With some composition implementation, visibility of issues can still be surfaced to the Claim so that owners are aware of trouble.
Beyond that, if there's something I'm missing on how to achieve the desired effect, I would be grateful for advice. Having two cases where the built-in backoff and re-queue facilities are inadequate shows me, I think, that we likely need some sort of facility in crossplane-runtime to adjust this behavior. For large control planes, this default behavior also impacts providers whose API resources are properly rate limited, since the rapid-fire retries of failed requests can then block working requests from succeeding due to rate limit behavior.
Related crossplane/crossplane-runtime#857
What problem are you facing?
I've observed use-cases where provider-terraform resources call out to APIs that do not properly implement rate limiting. The API owners get cranky about the number of retries made by Workspaces orchestrated by this provider since the retry mechanism is based on the Rate limit configuration defined in crossplane-runtime in addition to having no facility, currently, to adjust re-queuing of reconcile requests when there is an error.
What could help solve your problem?
Ideally, crossplane-runtime would support some mechanism to specify RequeueAfter or DoNotRequeue as part of Update/Create/Delete/Observe Result structs. This would be ignored when there is no error present but -- in the case of an error -- would be used to adjust the re-queue behavior of crossplane-runtime reconcile for these actions. I will likely file an issue there to see if this is at all possible, but I know this has been a bit of a challenge in the past. With this, we could expose a new MR field to accept a duration that will be fed to this configuration on error.
More ideally, downstream APIs would implement rate limiting and not care about this, but that is a different story :)
Less ideally, the Workspace reconciliation loop can be changed to conform to existing expected behavior, with an observability trade-off, in an opt-in fashion. I've implemented this pattern before in a native provider with a similarly un-rate-limited downstream API.
lastSyncFailedtimestamp instatus.atProvider. We can also tracklastSyncFailedConditionto capture the status.condition(s) data present when the failure occurredspec.forProvider.retryAfterfield (naming can be adjusted) that takes in a duration string declaring the desired minimum duration before the controller should attempt a full reconciliation of the resource. If null, existing behavior is preserved. We may also opt for an annotation-driven approach, to avoid polluting the schema with something that really should be solved in crossplane-runtimelastSyncFailedtimestamp against current time.The trade-off here is that MRs will report false readiness/sync states during the periods where they are skipping full reconciliation. With some composition implementation, visibility of issues can still be surfaced to the Claim so that owners are aware of trouble.
Beyond that, if there's something I'm missing on how to achieve the desired effect, I would be grateful for advice. Having two cases where the built-in backoff and re-queue facilities are inadequate shows me, I think, that we likely need some sort of facility in crossplane-runtime to adjust this behavior. For large control planes, this default behavior also impacts providers whose API resources are properly rate limited, since the rapid-fire retries of failed requests can then block working requests from succeeding due to rate limit behavior.
Related crossplane/crossplane-runtime#857