A recent week-long performance benchmarking session identified webhook component as a bottleneck.
Issues are particularly popping because of the configured UPDATE network validation hook. For some reason webhook
1: cannot handle a big number of concurrent calls, resulting in service outage
2: cannot truly recover from service outage
3: takes very long to handle a big number of concurrent requests (longer than it takes K8s API server to persist changes into etcd)
Until the internal bottlenecks are identified the UPDATE netvalidation hook shall be disabled.
Webhook works okay under low stress, so CREATE and DELETE workflows are still safe to be used.
A recent week-long performance benchmarking session identified webhook component as a bottleneck.
Issues are particularly popping because of the configured UPDATE network validation hook. For some reason webhook
1: cannot handle a big number of concurrent calls, resulting in service outage
2: cannot truly recover from service outage
3: takes very long to handle a big number of concurrent requests (longer than it takes K8s API server to persist changes into etcd)
Until the internal bottlenecks are identified the UPDATE netvalidation hook shall be disabled.
Webhook works okay under low stress, so CREATE and DELETE workflows are still safe to be used.