Schema isn't restored on replica after StatefullSet re-creation #1310
Description
Description
I'm facing an issue with recovering data after replica loss. After StatefullSet and PVCs corresponding to a replica are deleted, and then re-created by manually specifying spec.taskID in the ClickHouseInstallation yaml file, the schema isn't restored on the re-created replica.
Setup
Versions
clickhouse-operator: 0.22.2
clickhouse-server: 23.8
clickhouse-keeper: 23.8-alpine
Steps to reproduce
- Assuming clickhouse-keeper is deployed to the "clickhouse-recovery" namespace, and "openebs-hostpath" storage class is used for PVC
- Deploy cluster with 2 replicas and 1 shard using the following config:
chi.yaml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "recovery-test"
spec:
defaults:
templates:
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
podTemplate: clickhouse:23.8
configuration:
zookeeper:
nodes:
- host: clickhouse-keeper.clickhouse-recovery.svc
clusters:
- name: "ch-ha"
layout:
shardsCount: 1
replicasCount: 3
templates:
volumeClaimTemplates:
- name: data-volume-template
spec:
storageClassName: openebs-hostpath
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
- name: log-volume-template
spec:
storageClassName: openebs-hostpath
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
podTemplates:
- name: clickhouse:23.8
spec:
containers:
- name: clickhouse-pod
image: clickhouse/clickhouse-server:23.8
resources:
requests:
memory: "4Gi"
cpu: "1"
limits:
memory: "4Gi"
cpu: "1"Pods after deployment is :
NAME READY STATUS RESTARTS AGE
chi-recovery-test-ch-ha-0-0-0 2/2 Running 0 26m
chi-recovery-test-ch-ha-0-1-0 2/2 Running 0 45m
clickhouse-keeper-0 1/1 Running 0 27h
- Create a database, replicated table in the deployed cluster, and insert some data to it:
schema.sql
create database if not exists test_db on cluster 'ch-ha' engine Atomic;
CREATE TABLE if not exists test_db.test_table_local ON CLUSTER '{cluster}' (
`timestamp` DateTime CODEC(Delta(4), ZSTD(1)),
`key` String CODEC(ZSTD(1)),
) ENGINE = ReplicatedMergeTree(
'/clickhouse/{installation}/{cluster}/tables/{shard}/{database}/{table}',
'{replica}'
) PARTITION BY toDate(timestamp)
ORDER BY (key, timestamp);
CREATE TABLE IF NOT EXISTS test_db.test_table on CLUSTER '{cluster}' AS test_db.test_table_local ENGINE = Distributed(
'{cluster}',
test_db,
test_table_local,
xxh3(key)
);
INSERT INTO test_db.test_table_local(key,timestamp)
VALUES ('key1', now()), ('key2', now());- Delete StatefullSet and PVCs (to simulate an outage):
kubectl -n clickhouse-recovery delete sts chi-recovery-test-ch-ha-0-0
kubectl -n clickhouse-recovery delete pvc data-volume-template-chi-recovery-test-ch-ha-0-0-0 log-volume-template-chi-recovery-test-ch-ha-0-0-0
- Execute on the running replica ('chi-recovery-test-ch-ha-0-1'):
SYSTEM DROP REPLICA 'chi-recovery-test-ch-ha-0-0'- Change
spec.taskIDinClickHouseInstallationyaml file to some random string and apply it, forcing reconciliation - Wait for the
chi-recovery-test-ch-ha-0-0replica to be re-created - Check if
test_dbandtest_table_localare present onchi-recovery-test-ch-ha-0-0replica
Expected result
test_db and test_table_local are present on chi-recovery-test-ch-ha-0-0 replica, data in test_table_local is recovered from the live replica.
Actual result
Schema is not recreated, data isn't recovered
I've also observed these messages in the operator log:
I0125 08:55:09.956951 1 worker.go:898] migrateTables():No need to add tables on host 0 to shard 0 in cluster ch-ha
I0125 08:55:09.956964 1 worker.go:1038] includeHost():Include into cluster host 0 shard 0 cluster ch-ha
I0125 08:55:23.058351 1 cluster_schemer.go:238] HostCreateTables():Creating replicated objects at 0-1: [default test_db test_table test_table_local]
I0125 08:55:23.130474 1 cluster_schemer.go:245] HostCreateTables():Creating distributed objects at 0-1: [test_db]
It seems like instead of chi-recovery-test-ch-ha-0-0, operator tried to create schema on chi-recovery-test-ch-ha-0-1 ?....
Here is the full log:
Operator log
I0125 08:54:41.905914 1 controller.go:570] ENQUEUE new ReconcileCHI cmd=update for clickhouse-recovery/recovery-test
I0125 08:54:41.905988 1 worker.go:424] worker.go:424:updateCHI():start:clickhouse-recovery/recovery-test/force-recovery-17-54
I0125 08:54:41.906025 1 worker.go:432] updateCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:finalizer in place, proceed to reconcile cycle. CHI: clickhouse-recovery/recovery-test
I0125 08:54:41.906038 1 worker.go:469] Operator IPs to process CHI: recovery-test. Previous: 10.42.30.13 Cur: 10.42.30.13
I0125 08:54:41.906044 1 worker.go:478] Operator IPs are the same as on previous reconcile of the CHI: recovery-test
I0125 08:54:41.906050 1 worker.go:486] Operator is not just started. May not be clean restart
I0125 08:54:41.906733 1 worker.go:557] clickhouse-recovery/recovery-test:logCHI non-normalized yet (native) old start--------------------------------------------:
non-normalized yet (native) old
logCHI kind: ""
apiversion: ""
metadata:
name: recovery-test
generatename: ""
namespace: clickhouse-recovery
selflink: ""
uid: 49ca13ec-137a-4890-99fd-bedcda704e3e
resourceversion: "282625388"
generation: 1
creationtimestamp: "2024-01-25T08:39:15Z"
deletiontimestamp: null
deletiongraceperiodseconds: null
labels: {}
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"clickhouse.altinity.com/v1","kind":"ClickHouseInstallation","metadata":{"annotations":{},"name":"recovery-test","namespace":"clickhouse-recovery"},"spec":{"configuration":{"clusters":[{"layout":{"replicasCount":2,"shardsCount":1},"name":"ch-ha"}],"zookeeper":{"nodes":[{"host":"clickhouse-keeper.clickhouse-recovery.svc"}]}},"defaults":{"templates":{"dataVolumeClaimTemplate":"data-volume-template","logVolumeClaimTemplate":"log-volume-template","podTemplate":"clickhouse:23.8"}},"templates":{"podTemplates":[{"name":"clickhouse:23.8","spec":{"containers":[{"image":"clickhouse/clickhouse-server:23.8","name":"clickhouse-pod"}],"resources":{"limits":{"cpu":"1","memory":"4Gi"},"requests":{"cpu":"1","memory":"4Gi"}}}}],"volumeClaimTemplates":[{"name":"data-volume-template","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"openebs-hostpath"}},{"name":"log-volume-template","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"openebs-hostpath"}}]}}}
ownerreferences: []
finalizers:
- finalizer.clickhouseinstallation.altinity.com
clustername: ""
managedfields: []
spec:
defaults:
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
configuration:
zookeeper:
nodes:
- host: clickhouse-keeper.clickhouse-recovery.svc
clusters:
- name: ch-ha
layout:
shardsCount: 1
replicasCount: 2
templates:
podTemplates:
- name: clickhouse:23.8
spec:
volumes: []
initcontainers: []
containers:
- name: clickhouse-pod
image: clickhouse/clickhouse-server:23.8
command: []
args: []
workingdir: ""
ports: []
envfrom: []
env: []
resources:
limits: {}
requests: {}
volumemounts: []
volumedevices: []
livenessprobe: null
readinessprobe: null
startupprobe: null
lifecycle: null
terminationmessagepath: ""
terminationmessagepolicy: ""
imagepullpolicy: ""
securitycontext: null
stdin: false
stdinonce: false
tty: false
ephemeralcontainers: []
restartpolicy: ""
terminationgraceperiodseconds: null
activedeadlineseconds: null
dnspolicy: ""
nodeselector: {}
serviceaccountname: ""
deprecatedserviceaccount: ""
automountserviceaccounttoken: null
nodename: ""
hostnetwork: false
hostpid: false
hostipc: false
shareprocessnamespace: null
securitycontext: null
imagepullsecrets: []
hostname: ""
subdomain: ""
affinity: null
schedulername: ""
tolerations: []
hostaliases: []
priorityclassname: ""
priority: null
dnsconfig: null
readinessgates: []
runtimeclassname: null
enableservicelinks: null
preemptionpolicy: null
overhead: {}
topologyspreadconstraints: []
sethostnameasfqdn: null
volumeClaimTemplates:
- name: data-volume-template
storagemanagement: {}
spec:
accessmodes:
- ReadWriteOnce
selector: null
resources:
limits: {}
requests:
storage:
format: BinarySI
volumename: ""
storageclassname: openebs-hostpath
volumemode: null
datasource: null
- name: log-volume-template
storagemanagement: {}
spec:
accessmodes:
- ReadWriteOnce
selector: null
resources:
limits: {}
requests:
storage:
format: BinarySI
volumename: ""
storageclassname: openebs-hostpath
volumemode: null
datasource: null
end--------------------------------------------
I0125 08:54:41.907404 1 worker.go:557] clickhouse-recovery/recovery-test/force-recovery-17-54:logCHI non-normalized yet (native) new start--------------------------------------------:
non-normalized yet (native) new
logCHI kind: ClickHouseInstallation
apiversion: clickhouse.altinity.com/v1
metadata:
name: recovery-test
generatename: ""
namespace: clickhouse-recovery
selflink: ""
uid: 49ca13ec-137a-4890-99fd-bedcda704e3e
resourceversion: "282631348"
generation: 2
creationtimestamp: "2024-01-25T08:39:15Z"
deletiontimestamp: null
deletiongraceperiodseconds: null
labels: {}
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"clickhouse.altinity.com/v1","kind":"ClickHouseInstallation","metadata":{"annotations":{},"name":"recovery-test","namespace":"clickhouse-recovery"},"spec":{"configuration":{"clusters":[{"layout":{"replicasCount":2,"shardsCount":1},"name":"ch-ha"}],"zookeeper":{"nodes":[{"host":"clickhouse-keeper.clickhouse-recovery.svc"}]}},"defaults":{"templates":{"dataVolumeClaimTemplate":"data-volume-template","logVolumeClaimTemplate":"log-volume-template","podTemplate":"clickhouse:23.8"}},"taskID":"force-recovery-17-54","templates":{"podTemplates":[{"name":"clickhouse:23.8","spec":{"containers":[{"image":"clickhouse/clickhouse-server:23.8","name":"clickhouse-pod"}],"resources":{"limits":{"cpu":"1","memory":"4Gi"},"requests":{"cpu":"1","memory":"4Gi"}}}}],"volumeClaimTemplates":[{"name":"data-volume-template","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"openebs-hostpath"}},{"name":"log-volume-template","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"openebs-hostpath"}}]}}}
ownerreferences: []
finalizers:
- finalizer.clickhouseinstallation.altinity.com
clustername: ""
managedfields: []
spec:
taskID: force-recovery-17-54
defaults:
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
configuration:
zookeeper:
nodes:
- host: clickhouse-keeper.clickhouse-recovery.svc
clusters:
- name: ch-ha
layout:
shardsCount: 1
replicasCount: 2
templates:
podTemplates:
- name: clickhouse:23.8
spec:
volumes: []
initcontainers: []
containers:
- name: clickhouse-pod
image: clickhouse/clickhouse-server:23.8
command: []
args: []
workingdir: ""
ports: []
envfrom: []
env: []
resources:
limits: {}
requests: {}
volumemounts: []
volumedevices: []
livenessprobe: null
readinessprobe: null
startupprobe: null
lifecycle: null
terminationmessagepath: ""
terminationmessagepolicy: ""
imagepullpolicy: ""
securitycontext: null
stdin: false
stdinonce: false
tty: false
ephemeralcontainers: []
restartpolicy: ""
terminationgraceperiodseconds: null
activedeadlineseconds: null
dnspolicy: ""
nodeselector: {}
serviceaccountname: ""
deprecatedserviceaccount: ""
automountserviceaccounttoken: null
nodename: ""
hostnetwork: false
hostpid: false
hostipc: false
shareprocessnamespace: null
securitycontext: null
imagepullsecrets: []
hostname: ""
subdomain: ""
affinity: null
schedulername: ""
tolerations: []
hostaliases: []
priorityclassname: ""
priority: null
dnsconfig: null
readinessgates: []
runtimeclassname: null
enableservicelinks: null
preemptionpolicy: null
overhead: {}
topologyspreadconstraints: []
sethostnameasfqdn: null
volumeClaimTemplates:
- name: data-volume-template
storagemanagement: {}
spec:
accessmodes:
- ReadWriteOnce
selector: null
resources:
limits: {}
requests:
storage:
format: BinarySI
volumename: ""
storageclassname: openebs-hostpath
volumemode: null
datasource: null
- name: log-volume-template
storagemanagement: {}
spec:
accessmodes:
- ReadWriteOnce
selector: null
resources:
limits: {}
requests:
storage:
format: BinarySI
volumename: ""
storageclassname: openebs-hostpath
volumemode: null
datasource: null
end--------------------------------------------
I0125 08:54:41.907428 1 worker-chi-reconciler.go:53] worker-chi-reconciler.go:53:reconcileCHI():start:clickhouse-recovery/recovery-test/force-recovery-17-54
I0125 08:54:41.907448 1 worker-chi-reconciler.go:59] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:Changing OLD to Normalized COMPLETED: clickhouse-recovery/recovery-test
I0125 08:54:41.907466 1 worker-chi-reconciler.go:62] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:has ancestor, use it as a base for reconcile. CHI: clickhouse-recovery/recovery-test
I0125 08:54:41.907479 1 worker-chi-reconciler.go:69] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:Normalized OLD CHI: clickhouse-recovery/recovery-test
I0125 08:54:41.907496 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/6307791f-8ad9-4cae-bbda-7d2f15a7e741:Used templates count: 0
I0125 08:54:41.911567 1 worker.go:332] clickhouse-recovery/recovery-test/6307791f-8ad9-4cae-bbda-7d2f15a7e741:IPs of the CHI normalizer clickhouse-recovery/recovery-test: len: 1 [10.42.247.229]
I0125 08:54:41.911601 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/6307791f-8ad9-4cae-bbda-7d2f15a7e741:Used templates count: 0
I0125 08:54:41.912106 1 worker-chi-reconciler.go:72] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:Normalized NEW CHI: clickhouse-recovery/recovery-test
I0125 08:54:41.912127 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:41.916898 1 worker.go:332] clickhouse-recovery/recovery-test/force-recovery-17-54:IPs of the CHI normalizer clickhouse-recovery/recovery-test: len: 1 [10.42.247.229]
I0125 08:54:41.916911 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:41.917744 1 worker.go:557] clickhouse-recovery/recovery-test/6307791f-8ad9-4cae-bbda-7d2f15a7e741:logCHI normalized old start--------------------------------------------:
normalized old
logCHI kind: ClickHouseInstallation
apiversion: clickhouse.altinity.com/v1
metadata:
name: recovery-test
generatename: ""
namespace: clickhouse-recovery
selflink: ""
uid: 49ca13ec-137a-4890-99fd-bedcda704e3e
resourceversion: "282625367"
generation: 1
creationtimestamp: "2024-01-25T08:39:15Z"
deletiontimestamp: null
deletiongraceperiodseconds: null
labels: {}
annotations: {}
ownerreferences: []
finalizers:
- finalizer.clickhouseinstallation.altinity.com
clustername: ""
managedfields: []
spec:
taskID: 6307791f-8ad9-4cae-bbda-7d2f15a7e741
stop: "False"
troubleshoot: "False"
templating:
policy: manual
reconciling:
policy: unspecified
configMapPropagationTimeout: 10
cleanup:
unknownObjects:
statefulSet: Delete
pvc: Delete
configMap: Delete
service: Delete
reconcileFailedObjects:
statefulSet: Retain
pvc: Retain
configMap: Retain
service: Retain
defaults:
replicasUseFQDN: "False"
storageManagement: {}
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
configuration:
zookeeper:
nodes:
- host: clickhouse-keeper.clickhouse-recovery.svc
port: 2181
users:
clickhouse_operator/networks/ip: '[10.42.30.13]'
clickhouse_operator/password_sha256_hex: 716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448
clickhouse_operator/profile: clickhouse_operator
default/networks/host_regexp: (chi-recovery-test-[^.]+\d+-\d+|clickhouse\-recovery-test)\.clickhouse-recovery\.svc\.cluster\.local$
default/networks/ip: '[::1, 127.0.0.1, 10.42.247.229, 10.42.36.127]'
default/profile: default
default/quota: default
clusters:
- name: ch-ha
zookeeper:
nodes:
- host: clickhouse-keeper.clickhouse-recovery.svc
port: 2181
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
schemaPolicy:
replica: All
shard: All
layout:
shardsCount: 1
replicasCount: 2
shards:
- name: "0"
internalReplication: "True"
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
replicasCount: 2
replicas:
- name: 0-0
tcpPort: 9000
httpPort: 8123
interserverHTTPPort: 9009
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
- name: 0-1
tcpPort: 9000
httpPort: 8123
interserverHTTPPort: 9009
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
replicas:
- name: "0"
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
shardsCount: 1
shards:
- name: 0-0
tcpPort: 9000
httpPort: 8123
interserverHTTPPort: 9009
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
- name: "1"
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
shardsCount: 1
shards:
- name: 0-1
tcpPort: 9000
httpPort: 8123
interserverHTTPPort: 9009
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
templates:
podTemplates:
- name: clickhouse:23.8
spec:
volumes: []
initcontainers: []
containers:
- name: clickhouse-pod
image: clickhouse/clickhouse-server:23.8
command: []
args: []
workingdir: ""
ports: []
envfrom: []
env: []
resources:
limits: {}
requests: {}
volumemounts: []
volumedevices: []
livenessprobe: null
readinessprobe: null
startupprobe: null
lifecycle: null
terminationmessagepath: ""
terminationmessagepolicy: ""
imagepullpolicy: ""
securitycontext: null
stdin: false
stdinonce: false
tty: false
ephemeralcontainers: []
restartpolicy: ""
terminationgraceperiodseconds: null
activedeadlineseconds: null
dnspolicy: ""
nodeselector: {}
serviceaccountname: ""
deprecatedserviceaccount: ""
automountserviceaccounttoken: null
nodename: ""
hostnetwork: false
hostpid: false
hostipc: false
shareprocessnamespace: null
securitycontext: null
imagepullsecrets: []
hostname: ""
subdomain: ""
affinity: null
schedulername: ""
tolerations: []
hostaliases: []
priorityclassname: ""
priority: null
dnsconfig: null
readinessgates: []
runtimeclassname: null
enableservicelinks: null
preemptionpolicy: null
overhead: {}
topologyspreadconstraints: []
sethostnameasfqdn: null
volumeClaimTemplates:
- name: data-volume-template
storagemanagement: {}
spec:
accessmodes:
- ReadWriteOnce
selector: null
resources:
limits: {}
requests:
storage:
format: BinarySI
volumename: ""
storageclassname: openebs-hostpath
volumemode: null
datasource: null
- name: log-volume-template
storagemanagement: {}
spec:
accessmodes:
- ReadWriteOnce
selector: null
resources:
limits: {}
requests:
storage:
format: BinarySI
volumename: ""
storageclassname: openebs-hostpath
volumemode: null
datasource: null
podtemplatesindex: {}
volumeclaimtemplatesindex: {}
end--------------------------------------------
I0125 08:54:41.918474 1 worker.go:557] clickhouse-recovery/recovery-test/force-recovery-17-54:logCHI normalized new start--------------------------------------------:
normalized new
logCHI kind: ClickHouseInstallation
apiversion: clickhouse.altinity.com/v1
metadata:
name: recovery-test
generatename: ""
namespace: clickhouse-recovery
selflink: ""
uid: 49ca13ec-137a-4890-99fd-bedcda704e3e
resourceversion: "282631348"
generation: 2
creationtimestamp: "2024-01-25T08:39:15Z"
deletiontimestamp: null
deletiongraceperiodseconds: null
labels: {}
annotations: {}
ownerreferences: []
finalizers:
- finalizer.clickhouseinstallation.altinity.com
clustername: ""
managedfields: []
spec:
taskID: force-recovery-17-54
stop: "False"
troubleshoot: "False"
templating:
policy: manual
reconciling:
policy: unspecified
configMapPropagationTimeout: 10
cleanup:
unknownObjects:
statefulSet: Delete
pvc: Delete
configMap: Delete
service: Delete
reconcileFailedObjects:
statefulSet: Retain
pvc: Retain
configMap: Retain
service: Retain
defaults:
replicasUseFQDN: "False"
storageManagement: {}
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
configuration:
zookeeper:
nodes:
- host: clickhouse-keeper.clickhouse-recovery.svc
port: 2181
users:
clickhouse_operator/networks/ip: '[10.42.30.13]'
clickhouse_operator/password_sha256_hex: 716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448
clickhouse_operator/profile: clickhouse_operator
default/networks/host_regexp: (chi-recovery-test-[^.]+\d+-\d+|clickhouse\-recovery-test)\.clickhouse-recovery\.svc\.cluster\.local$
default/networks/ip: '[::1, 127.0.0.1, 10.42.247.229]'
default/profile: default
default/quota: default
clusters:
- name: ch-ha
zookeeper:
nodes:
- host: clickhouse-keeper.clickhouse-recovery.svc
port: 2181
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
schemaPolicy:
replica: All
shard: All
layout:
shardsCount: 1
replicasCount: 2
shards:
- name: "0"
internalReplication: "True"
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
replicasCount: 2
replicas:
- name: 0-0
tcpPort: 9000
httpPort: 8123
interserverHTTPPort: 9009
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
- name: 0-1
tcpPort: 9000
httpPort: 8123
interserverHTTPPort: 9009
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
replicas:
- name: "0"
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
shardsCount: 1
shards:
- name: 0-0
tcpPort: 9000
httpPort: 8123
interserverHTTPPort: 9009
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
- name: "1"
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
shardsCount: 1
shards:
- name: 0-1
tcpPort: 9000
httpPort: 8123
interserverHTTPPort: 9009
templates:
podTemplate: clickhouse:23.8
dataVolumeClaimTemplate: data-volume-template
logVolumeClaimTemplate: log-volume-template
templates:
podTemplates:
- name: clickhouse:23.8
spec:
volumes: []
initcontainers: []
containers:
- name: clickhouse-pod
image: clickhouse/clickhouse-server:23.8
command: []
args: []
workingdir: ""
ports: []
envfrom: []
env: []
resources:
limits: {}
requests: {}
volumemounts: []
volumedevices: []
livenessprobe: null
readinessprobe: null
startupprobe: null
lifecycle: null
terminationmessagepath: ""
terminationmessagepolicy: ""
imagepullpolicy: ""
securitycontext: null
stdin: false
stdinonce: false
tty: false
ephemeralcontainers: []
restartpolicy: ""
terminationgraceperiodseconds: null
activedeadlineseconds: null
dnspolicy: ""
nodeselector: {}
serviceaccountname: ""
deprecatedserviceaccount: ""
automountserviceaccounttoken: null
nodename: ""
hostnetwork: false
hostpid: false
hostipc: false
shareprocessnamespace: null
securitycontext: null
imagepullsecrets: []
hostname: ""
subdomain: ""
affinity: null
schedulername: ""
tolerations: []
hostaliases: []
priorityclassname: ""
priority: null
dnsconfig: null
readinessgates: []
runtimeclassname: null
enableservicelinks: null
preemptionpolicy: null
overhead: {}
topologyspreadconstraints: []
sethostnameasfqdn: null
volumeClaimTemplates:
- name: data-volume-template
storagemanagement: {}
spec:
accessmodes:
- ReadWriteOnce
selector: null
resources:
limits: {}
requests:
storage:
format: BinarySI
volumename: ""
storageclassname: openebs-hostpath
volumemode: null
datasource: null
- name: log-volume-template
storagemanagement: {}
spec:
accessmodes:
- ReadWriteOnce
selector: null
resources:
limits: {}
requests:
storage:
format: BinarySI
volumename: ""
storageclassname: openebs-hostpath
volumemode: null
datasource: null
podtemplatesindex: {}
volumeclaimtemplatesindex: {}
end--------------------------------------------
I0125 08:54:41.918823 1 worker.go:567] ActionPlan start---------------------------------------------:
AP item start -------------------------
removed spec items: 1
ap item path [0]:'.Configuration.Users["default/networks/ip"].vector[3]'
ap item value[0]:'"10.42.36.127"'
AP item end -------------------------
AP item start -------------------------
modified spec items: 1
ap item path [0]:'.TaskID'
ap item value[0]:'"force-recovery-17-54"'
AP item end -------------------------
ActionPlan end---------------------------------------------
I0125 08:54:41.918851 1 worker-chi-reconciler.go:83] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:ActionPlan has actions - continue reconcile
I0125 08:54:41.935295 1 worker.go:651] markReconcileStart():clickhouse-recovery/recovery-test/force-recovery-17-54:reconcile started, task id: force-recovery-17-54
I0125 08:54:42.512827 1 worker.go:806] FOUND host: ns:clickhouse-recovery|chi:recovery-test|clu:ch-ha|sha:0|rep:0|host:0-0
I0125 08:54:42.512844 1 worker.go:806] FOUND host: ns:clickhouse-recovery|chi:recovery-test|clu:ch-ha|sha:0|rep:1|host:0-1
I0125 08:54:42.512898 1 worker.go:834] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I0125 08:54:42.710480 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-configd
I0125 08:54:43.517896 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-usersd
I0125 08:54:44.128418 1 creator.go:113] CreateServiceCluster():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/cluster-recovery-test-ch-ha
I0125 08:54:44.134928 1 worker-chi-reconciler.go:716] PDB updated clickhouse-recovery/recovery-test-ch-ha
I0125 08:54:44.134949 1 worker-chi-reconciler.go:492] not found ReconcileShardsAndHostsOptionsCtxKey, use empty opts
I0125 08:54:44.134955 1 worker-chi-reconciler.go:506] starting first shard separately
I0125 08:54:44.134992 1 creator.go:136] CreateServiceShard():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/shard-recovery-test-ch-ha-0
I0125 08:54:44.135043 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local]
E0125 08:54:44.138380 1 connection.go:145] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local:8123/) doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local on 10.43.0.10:53: no such host for SQL: SELECT version()
W0125 08:54:44.138392 1 cluster.go:91] QueryAny():FAILED to run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local on 10.43.0.10:53: no such host
E0125 08:54:44.138400 1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local]
W0125 08:54:44.138414 1 worker-chi-reconciler.go:308] getHostClickHouseVersion():Failed to get ClickHouse version on host: 0-0
I0125 08:54:44.138425 1 worker-chi-reconciler.go:621] reconcileHost():Reconcile Host start. Host: 0-0 ClickHouse version running: failed to query
I0125 08:54:44.724494 1 creator.go:589] getPodTemplate():clickhouse-recovery/recovery-test/force-recovery-17-54:statefulSet chi-recovery-test-ch-ha-0-0 use custom template: clickhouse:23.8
I0125 08:54:44.724647 1 creator.go:575] setupLogContainer():clickhouse-recovery/recovery-test/force-recovery-17-54:add log container for statefulSet chi-recovery-test-ch-ha-0-0
I0125 08:54:44.725548 1 worker.go:984] worker.go:984:excludeHost():start:exclude host start
I0125 08:54:44.725577 1 worker.go:120] shouldForceRestartHost():Host is new, no restart applicable. Host: 0-0
I0125 08:54:44.725585 1 worker.go:1125] shouldExcludeHost():Host is new, no need to exclude host 0 shard 0 cluster ch-ha
I0125 08:54:44.725589 1 worker.go:988] worker.go:985:excludeHost():end:exclude host end
I0125 08:54:44.725597 1 worker.go:1002] worker.go:1002:completeQueries():start:complete queries start
I0125 08:54:44.725611 1 worker.go:1174] shouldWaitQueries():No need to wait for queries to complete, host is a new one host 0 shard 0 cluster ch-ha
I0125 08:54:44.725615 1 worker.go:1009] worker.go:1003:completeQueries():end:complete queries end
I0125 08:54:44.910691 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-0
I0125 08:54:46.310927 1 worker-chi-reconciler.go:352] worker-chi-reconciler.go:352:reconcileHostStatefulSet():start:reconcile StatefulSet start
I0125 08:54:46.312591 1 worker-chi-reconciler.go:358] reconcileHostStatefulSet():Reconcile host 0-0. ClickHouse version: not applicable
I0125 08:54:46.312612 1 worker.go:120] shouldForceRestartHost():Host is new, no restart applicable. Host: 0-0
I0125 08:54:46.312625 1 worker-chi-reconciler.go:371] reconcileHostStatefulSet():Reconcile host 0-0. Reconcile StatefulSet
I0125 08:54:46.312703 1 creator.go:589] getPodTemplate():clickhouse-recovery/recovery-test/force-recovery-17-54:statefulSet chi-recovery-test-ch-ha-0-0 use custom template: clickhouse:23.8
I0125 08:54:46.312791 1 creator.go:575] setupLogContainer():clickhouse-recovery/recovery-test/force-recovery-17-54:add log container for statefulSet chi-recovery-test-ch-ha-0-0
I0125 08:54:46.313602 1 worker.go:1539] createStatefulSet():Create StatefulSet clickhouse-recovery/chi-recovery-test-ch-ha-0-0 - started
I0125 08:54:46.927327 1 creator.go:34] createStatefulSet()
I0125 08:54:46.927335 1 creator.go:43] Create StatefulSet clickhouse-recovery/chi-recovery-test-ch-ha-0-0
I0125 08:54:46.983937 1 worker.go:259] processReconcilePod():clickhouse-recovery/chi-recovery-test-ch-ha-0-0-0:Add Pod. clickhouse-recovery/chi-recovery-test-ch-ha-0-0-0
I0125 08:54:52.521502 1 poller.go:213] pollStatefulSet():clickhouse-recovery/chi-recovery-test-ch-ha-0-0:OK :ObservedGeneration:1 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-recovery-test-ch-ha-0-0-d7556485c UpdateRevision:chi-recovery-test-ch-ha-0-0-d7556485c
I0125 08:54:56.872298 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:56.884734 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:56.885120 1 worker.go:380] clickhouse-recovery/recovery-test/force-recovery-17-54:updating endpoints for CHI-1 recovery-test
I0125 08:54:56.888017 1 worker.go:382] clickhouse-recovery/recovery-test/force-recovery-17-54:IPs of the CHI-1 update endpoints clickhouse-recovery/recovery-test: len: 2 [10.42.36.80 10.42.247.229]
I0125 08:54:56.890261 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:56.890678 1 worker.go:386] clickhouse-recovery/recovery-test/force-recovery-17-54:Update users IPS-1
I0125 08:54:56.898916 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-usersd
I0125 08:55:07.542233 1 poller.go:213] pollStatefulSet():clickhouse-recovery/chi-recovery-test-ch-ha-0-0:OK :ObservedGeneration:1 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-recovery-test-ch-ha-0-0-d7556485c UpdateRevision:chi-recovery-test-ch-ha-0-0-d7556485c
I0125 08:55:07.561262 1 worker.go:1556] createStatefulSet():Create StatefulSet clickhouse-recovery/chi-recovery-test-ch-ha-0-0 - completed
I0125 08:55:07.590074 1 worker-chi-reconciler.go:383] worker-chi-reconciler.go:353:reconcileHostStatefulSet():end:reconcile StatefulSet end
I0125 08:55:09.146809 1 creator.go:160] CreateServiceHost():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/chi-recovery-test-ch-ha-0-0 for Set chi-recovery-test-ch-ha-0-0
I0125 08:55:09.342925 1 worker.go:1407] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:Update Service clickhouse-recovery/chi-recovery-test-ch-ha-0-0
I0125 08:55:09.956874 1 worker-chi-reconciler.go:399] reconcileHostService():DONE Reconcile service of the host 0-0.
I0125 08:55:09.956951 1 worker.go:898] migrateTables():No need to add tables on host 0 to shard 0 in cluster ch-ha
I0125 08:55:09.956964 1 worker.go:1038] includeHost():Include into cluster host 0 shard 0 cluster ch-ha
I0125 08:55:09.956972 1 worker.go:834] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I0125 08:55:10.141671 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-configd
I0125 08:55:11.163052 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:11.177639 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:11.178051 1 worker.go:380] clickhouse-recovery/recovery-test/force-recovery-17-54:updating endpoints for CHI-1 recovery-test
I0125 08:55:11.347896 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local]
I0125 08:55:11.349467 1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Get ClickHouse version on host: 0-0 version: 23.8.9.54
I0125 08:55:11.349493 1 worker-chi-reconciler.go:684] reconcileHost():Reconcile Host completed. Host: 0-0 ClickHouse version running: 23.8.9.54
I0125 08:55:11.941576 1 worker.go:382] clickhouse-recovery/recovery-test/force-recovery-17-54:IPs of the CHI-1 update endpoints clickhouse-recovery/recovery-test: len: 2 [10.42.36.80 10.42.247.229]
I0125 08:55:11.944386 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:11.944784 1 worker.go:386] clickhouse-recovery/recovery-test/force-recovery-17-54:Update users IPS-1
I0125 08:55:12.345129 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-usersd
I0125 08:55:12.557245 1 worker-chi-reconciler.go:700] reconcileHost():ProgressHostsCompleted: 1 of 2
I0125 08:55:13.760509 1 creator.go:58] CreateServiceCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/clickhouse-recovery-test
I0125 08:55:13.760710 1 worker.go:1359] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:reuse Port 8123 values
I0125 08:55:13.760730 1 worker.go:1359] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:reuse Port 9000 values
I0125 08:55:14.141780 1 worker.go:1407] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:Update Service clickhouse-recovery/clickhouse-recovery-test
I0125 08:55:14.957647 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:14.962303 1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: 23.8.9.54
I0125 08:55:14.962319 1 worker-chi-reconciler.go:621] reconcileHost():Reconcile Host start. Host: 0-1 ClickHouse version running: 23.8.9.54
I0125 08:55:15.559444 1 creator.go:589] getPodTemplate():clickhouse-recovery/recovery-test/force-recovery-17-54:statefulSet chi-recovery-test-ch-ha-0-1 use custom template: clickhouse:23.8
I0125 08:55:15.559513 1 creator.go:575] setupLogContainer():clickhouse-recovery/recovery-test/force-recovery-17-54:add log container for statefulSet chi-recovery-test-ch-ha-0-1
I0125 08:55:15.560177 1 worker.go:1508] getObjectStatusFromMetas():clickhouse-recovery/chi-recovery-test-ch-ha-0-1:cur and new objects are equal based on object version label. Update of the object is not required. Object: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:15.560193 1 worker.go:984] worker.go:984:excludeHost():start:exclude host start
I0125 08:55:15.741564 1 worker.go:152] shouldForceRestartHost():Host restart is not required. Host: 0-1
I0125 08:55:15.741588 1 worker.go:1130] shouldExcludeHost():Host is the same, would not be updated, no need to exclude host 1 shard 0 cluster ch-ha
I0125 08:55:15.741595 1 worker.go:988] worker.go:985:excludeHost():end:exclude host end
I0125 08:55:15.741607 1 worker.go:1002] worker.go:1002:completeQueries():start:complete queries start
I0125 08:55:15.741624 1 worker.go:1179] shouldWaitQueries():Will wait for queries to complete according to CHOp config reconcile.host.wait.queries setting, host is not yet in the cluster host 1 shard 0 cluster ch-ha
I0125 08:55:15.741686 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:15.746300 1 poller.go:307] pollHostContext():clickhouse-recovery/0-1-OK
I0125 08:55:15.746313 1 worker.go:1006] worker.go:1003:completeQueries():end:complete queries end
I0125 08:55:15.942155 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-1
I0125 08:55:18.948947 1 worker-chi-reconciler.go:352] worker-chi-reconciler.go:352:reconcileHostStatefulSet():start:reconcile StatefulSet start
I0125 08:55:18.948995 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:19.043603 1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: 23.8.9.54
I0125 08:55:19.045890 1 worker-chi-reconciler.go:358] reconcileHostStatefulSet():Reconcile host 0-1. ClickHouse version: 23.8.9.54
I0125 08:55:19.140999 1 worker.go:152] shouldForceRestartHost():Host restart is not required. Host: 0-1
I0125 08:55:19.141033 1 worker-chi-reconciler.go:371] reconcileHostStatefulSet():Reconcile host 0-1. Reconcile StatefulSet
I0125 08:55:19.141126 1 creator.go:589] getPodTemplate():clickhouse-recovery/recovery-test/force-recovery-17-54:statefulSet chi-recovery-test-ch-ha-0-1 use custom template: clickhouse:23.8
I0125 08:55:19.141222 1 creator.go:575] setupLogContainer():clickhouse-recovery/recovery-test/force-recovery-17-54:add log container for statefulSet chi-recovery-test-ch-ha-0-1
I0125 08:55:19.142000 1 worker.go:1508] getObjectStatusFromMetas():clickhouse-recovery/chi-recovery-test-ch-ha-0-1:cur and new objects are equal based on object version label. Update of the object is not required. Object: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:19.142070 1 worker-chi-reconciler.go:383] worker-chi-reconciler.go:353:reconcileHostStatefulSet():end:reconcile StatefulSet end
I0125 08:55:21.544663 1 creator.go:160] CreateServiceHost():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/chi-recovery-test-ch-ha-0-1 for Set chi-recovery-test-ch-ha-0-1
I0125 08:55:21.741837 1 worker.go:1407] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:Update Service clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:22.357605 1 worker-chi-reconciler.go:399] reconcileHostService():DONE Reconcile service of the host 0-1.
I0125 08:55:22.357676 1 worker.go:915] migrateTables():Adding tables on shard/host:0/1 cluster:ch-ha
I0125 08:55:22.955996 1 cluster_schemer.go:228] cluster_schemer.go:228:HostCreateTables():start:Migrating schema objects to host 0-1
I0125 08:55:22.956073 1 cluster_schemer.go:128] shouldCreateReplicatedObjects():SchemaPolicy.Shard says we need replicated objects. Should create replicated objects for the shard: [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.956107 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.965244 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.971506 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.975558 1 cluster_schemer.go:58] shouldCreateDistributedObjects():Should create distributed objects in the cluster: [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.975590 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:23.025180 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:23.054752 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:23.058351 1 cluster_schemer.go:238] HostCreateTables():Creating replicated objects at 0-1: [default test_db test_table test_table_local]
I0125 08:55:23.130474 1 cluster_schemer.go:245] HostCreateTables():Creating distributed objects at 0-1: [test_db]
I0125 08:55:23.130869 1 cluster_schemer.go:257] cluster_schemer.go:229:HostCreateTables():end:Migrating schema objects to host 0-1
I0125 08:55:23.130900 1 worker.go:923] migrateTables():Tables added successfully on shard/host:0/1 cluster:ch-ha
I0125 08:55:23.559150 1 worker.go:1038] includeHost():Include into cluster host 1 shard 0 cluster ch-ha
I0125 08:55:23.559168 1 worker.go:834] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I0125 08:55:23.741635 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-configd
I0125 08:55:24.541636 1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:24.542801 1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: 23.8.9.54
I0125 08:55:24.542818 1 worker-chi-reconciler.go:684] reconcileHost():Reconcile Host completed. Host: 0-1 ClickHouse version running: 23.8.9.54
I0125 08:55:25.158705 1 worker-chi-reconciler.go:700] reconcileHost():ProgressHostsCompleted: 2 of 2
I0125 08:55:25.757534 1 worker-chi-reconciler.go:519] Starting rest of shards on workers: 1
I0125 08:55:25.942327 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-configd
I0125 08:55:26.559559 1 worker-chi-reconciler.go:117] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:remove items scheduled for deletion
I0125 08:55:27.157667 1 worker-deleter.go:38] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:Failed to reconcile objects:
I0125 08:55:27.157694 1 worker-deleter.go:39] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:Reconciled objects:
ConfigMap: clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-0
ConfigMap: clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-1
ConfigMap: clickhouse-recovery/chi-recovery-test-common-configd
ConfigMap: clickhouse-recovery/chi-recovery-test-common-usersd
PDB: clickhouse-recovery/recovery-test-ch-ha
StatefulSet: clickhouse-recovery/chi-recovery-test-ch-ha-0-0
StatefulSet: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
PVC: clickhouse-recovery/data-volume-template-chi-recovery-test-ch-ha-0-0-0
PVC: clickhouse-recovery/log-volume-template-chi-recovery-test-ch-ha-0-0-0
PVC: clickhouse-recovery/data-volume-template-chi-recovery-test-ch-ha-0-1-0
PVC: clickhouse-recovery/log-volume-template-chi-recovery-test-ch-ha-0-1-0
Service: clickhouse-recovery/chi-recovery-test-ch-ha-0-0
Service: clickhouse-recovery/clickhouse-recovery-test
Service: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:27.943225 1 worker-deleter.go:42] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:Existing objects:
ConfigMap: clickhouse-recovery/chi-recovery-test-common-usersd
ConfigMap: clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-0
ConfigMap: clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-1
ConfigMap: clickhouse-recovery/chi-recovery-test-common-configd
Service: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
Service: clickhouse-recovery/clickhouse-recovery-test
Service: clickhouse-recovery/chi-recovery-test-ch-ha-0-0
PVC: clickhouse-recovery/data-volume-template-chi-recovery-test-ch-ha-0-0-0
PVC: clickhouse-recovery/data-volume-template-chi-recovery-test-ch-ha-0-1-0
PVC: clickhouse-recovery/log-volume-template-chi-recovery-test-ch-ha-0-0-0
PVC: clickhouse-recovery/log-volume-template-chi-recovery-test-ch-ha-0-1-0
PDB: clickhouse-recovery/recovery-test-ch-ha
StatefulSet: clickhouse-recovery/chi-recovery-test-ch-ha-0-0
StatefulSet: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:27.943263 1 worker-deleter.go:44] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:Non-reconciled objects:
I0125 08:55:27.943273 1 worker-deleter.go:54] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:remove items scheduled for deletion
I0125 08:55:28.552630 1 worker-deleter.go:66] worker-deleter.go:66:dropReplicas():start:clickhouse-recovery/recovery-test/force-recovery-17-54:drop replicas based on AP
I0125 08:55:28.552660 1 worker-deleter.go:78] worker-deleter.go:78:dropReplicas():end:clickhouse-recovery/recovery-test/force-recovery-17-54:processed replicas: 0
I0125 08:55:28.552675 1 worker.go:628] addCHIToMonitoring():clickhouse-recovery/recovery-test/force-recovery-17-54:add CHI to monitoring
I0125 08:55:29.158949 1 controller.go:615] OK update watch (clickhouse-recovery/recovery-test): {"namespace":"clickhouse-recovery","name":"recovery-test","clusters":[{"name":"ch-ha","hosts":[{"name":"0-0","hostname":"chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local","tcpPort":9000,"httpPort":8123},{"name":"0-1","hostname":"chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local","tcpPort":9000,"httpPort":8123}]}]}
I0125 08:55:29.161007 1 worker.go:588] clickhouse-recovery/recovery-test/force-recovery-17-54:all IP addresses are in place
I0125 08:55:29.163319 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:29.163678 1 worker.go:663] clickhouse-recovery/recovery-test/force-recovery-17-54:updating endpoints for CHI-2 recovery-test
I0125 08:55:29.541088 1 worker.go:665] clickhouse-recovery/recovery-test/force-recovery-17-54:IPs of the CHI-2 finalize reconcile clickhouse-recovery/recovery-test: len: 2 [10.42.36.80 10.42.247.229]
I0125 08:55:29.543767 1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:29.544242 1 worker.go:669] clickhouse-recovery/recovery-test/force-recovery-17-54:Update users IPS-2
I0125 08:55:29.741928 1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-usersd
I0125 08:55:30.757498 1 worker.go:693] finalizeReconcileAndMarkCompleted():clickhouse-recovery/recovery-test/force-recovery-17-54:reconcile completed successfully, task id: force-recovery-17-54
I0125 08:55:31.358074 1 worker-chi-reconciler.go:128] worker-chi-reconciler.go:54:reconcileCHI():end:clickhouse-recovery/recovery-test/force-recovery-17-54
I0125 08:55:31.358095 1 worker.go:462] worker.go:425:updateCHI():end:clickhouse-recovery/recovery-test/force-recovery-17-54
Metadata
Metadata
Assignees
Labels
No labels