Filtering Event of Informers After Watched Resource Update

[csviri]

Problem Statement

In case we during the reconciliation update a dependent resource, we receive a new event regarding the change of the dependent resource from the informer. In general this even can be skipped since we know that the state of the target resource is already the desired one. The way to do this would be compare resourceVersion of the resource of the received event with the resource version of a cached resource (not the current informer cache).

Proposed Solution

One relatively transparent way would be to propagate the resourceVersion to the informer to cache it. When we receive the next event compare the resources versions, if are the same just skip the event. But either is the same or not, instantly remove the cached resourceVersion from the cache. This way we don't cache too long the version, and don't have to specially handle the cleanup on resource delete event.

Note that dependent resources could do the propagation automatically, while on low level API this needs to be done explicitly by the implementation.

[scrocquesel]

Regarding concurrency issue would a semaphore mechnism allows to update the resource while blocking the evensource until the resourceVersion is propagated. ? The drawback is that the informer will not be able to process new events, so maybe it can retain events concerning only the resource being edited.

This should have no impact because we are still in the reconcile loop and imho, we should return the same version of the dependent resource that raise the reconciliation. That is to say, two different calls in the same reconcile loop to context.getSecondaryResource shall return the same resource version.

[csviri]

Regarding concurrency issue would a semaphore mechnism allows to update the resource while blocking the evensource until the resourceVersion is propagated. ? The drawback is that the informer will not be able to process new events, so maybe it can retain events concerning only the resource being edited

This is not really feasible, because there might be other events, meanwhile, and the controller might fail or other errors might happen etc, that would need special care etc. Also the event can be lost in some corner cases.

I think the proposed solution is good enough, btw we already do something similar with the custom resource, that seems to be work fine.

That is to say, two different calls in the same reconcile loop to context.getSecondaryResource shall return the same resource version.

Ideally yes, there should be kinda "snapshot isolation", but its not that big issue if doesn't imho, also it can be handled on code level, plus usually the resource only read once. I pretty sure that controller-runtime is not doing this either.
However, note that there is no concept of time here, we just redo the reconciliation if something changed.. But eventually we are consistent.

[csviri]

The linked implementation does the job, but not always, especially locally the event is sometimes arrived before the temporal cached is updated. I guess this is the best we can do ATM.

[csviri]

An additional improvement could be that have this as a two phase create/update. Like mark that we will update resource so, no events are processed from that point. Maybe record the events coming meanwhile. And after the create/update make a second call to evaluate the situation. That will solve the concurrency issues, but will complicate the implementation.

[csviri]

Currently the delete is also missing, will think of that too.