metrics exporter lose monitored ClickHouse instances list and stop publish metrics, after restart metrics-exporter container inside clickhouse-operator pod

[Slach]

Steps to reproduce

• install clickhouse-operator
• install any kind: ClickhouseInstallation from chi-examples
• reboot metrics-exporter via

kubectl exec -n $( kubectl get pods --all-namespaces | grep clickhouse-operator | awk '{print $1 " " $2}') -c metrics-exporter reboot

• run port-forward to metrics exporter 8888

kubectl -n $( kubectl get svc --all-namespaces | grep clickhouse-operator-metrics | awk '{print $1}') port-forward service/clickhouse-operator-metrics 8888

• open http://localhost:8888/chi

Expected results

json array with current running clickhouse instances

Actual results

empty json

This state also happens in following situations:

• OOP Kill clickhouse-operator/metrics-exporter container
• reschedule clickhouse-operator/metrics-exporter
• whole restart kubernetes node / cluster

Solution proposal

• simple add retries and timeout options to on MakeRESTCall in file https://github.com/Altinity/clickhouse-operator/blob/master/pkg/apis/metrics/rest.go
• add retries and logging to https://github.com/Altinity/clickhouse-operator/blob/master/pkg/controller/chi/controller.go on updateWatchAsync

[alex-zaitsev]

Thanks, we know this. Needs to be fixed

[sunsingerus]

Please, check new feature

[sunsingerus]

fix available in 0.9.1 branch

[alex-zaitsev]

Fixed in https://github.com/Altinity/clickhouse-operator/releases/tag/0.9.1