cluster takes forever to stop because of the connection refused error

[Elmo33]

I am executing the autotest to check cluster stop/resume functionality, and whenver I try to stop the cluster, it takes up to 15/20 minutes to resume because of the connection refused error. issue can be easily reproduced, created cluster,waited for all the checks, stopped.

E0215 20:09:10.497764       1 connection.go:145] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local:8123/) doRequest: transport failed to send a request to ClickHouse: dial tcp 172.20.70.65:8123: connect: connection refused for SQL: SELECT version()
W0215 20:09:10.497793       1 cluster.go:91] QueryAny():FAILED to run query on: chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local of [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp 172.20.70.65:8123: connect: connection refused
E0215 20:09:10.498045       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local]
W0215 20:09:10.498083       1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Failed to get ClickHouse version on host: 0-1
W0215 20:09:10.498101       1 worker-chi-reconciler.go:334] func1():Host is NOT alive: 0-1 
I0215 20:09:10.498122       1 poller.go:170] Poll():qa-release-ui/0-1:WAIT:qa-release-ui/0-1
I0215 20:09:15.498819       1 cluster.go:84] Run query on: chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local of [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local]
E0215 20:09:15.504775       1 connection.go:145] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local:8123/) doRequest: transport failed to send a request to ClickHouse: dial tcp 172.20.70.65:8123: connect: connection refused for SQL: SELECT version()
W0215 20:09:15.504804       1 cluster.go:91] QueryAny():FAILED to run query on: chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local of [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp 172.20.70.65:8123: connect: connection refused
E0215 20:09:15.504823       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local]
W0215 20:09:15.504848       1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Failed to get ClickHouse version on host: 0-1
W0215 20:09:15.504911       1 worker-chi-reconciler.go:334] func1():Host is NOT alive: 0-1 
I0215 20:09:15.504991       1 poller.go:170] Poll():qa-release-ui/0-1:WAIT:qa-release-ui/0-1
I0215 20:09:20.505618       1 cluster.go:84] Run query on: chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local of [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local]
E0215 20:09:20.512542       1 connection.go:145] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local:8123/) doRequest: transport failed to send a request to ClickHouse: dial tcp 172.20.70.65:8123: connect: connection refused for SQL: SELECT version()
W0215 20:09:20.512586       1 cluster.go:91] QueryAny():FAILED to run query on: chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local of [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp 172.20.70.65:8123: connect: connection refused
E0215 20:09:20.512605       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local]
W0215 20:09:20.512635       1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Failed to get ClickHouse version on host: 0-1
W0215 20:09:20.512651       1 worker-chi-reconciler.go:334] func1():Host is NOT alive: 0-1 
I0215 20:09:20.512675       1 poller.go:170] Poll():qa-release-ui/0-1:WAIT:qa-release-ui/0-1
I0215 20:09:25.513141       1 cluster.go:84] Run query on: chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local of [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local]
E0215 20:09:25.519074       1 connection.go:145] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local:8123/) doRequest: transport failed to send a request to ClickHouse: dial tcp 172.20.70.65:8123: connect: connection refused for SQL: SELECT version()
W0215 20:09:25.519105       1 cluster.go:91] QueryAny():FAILED to run query on: chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local of [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp 172.20.70.65:8123: connect: connection refused
E0215 20:09:25.519122       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-testcluster-testcluster-0-1.qa-release-ui.svc.cluster.local]
W0215 20:09:25.519148       1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Failed to get ClickHouse version on host: 0-1
W0215 20:09:25.519169       1 worker-chi-reconciler.go:334] func1():Host is NOT alive: 0-1 

[alex-zaitsev]

Fixed in 0.23.3

[alex-zaitsev]

Fixed in https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.23.3