TLS Connection Issue: Unable to Establish Secure Connection Between ClickHouse Operator and ClickHouse Keeper

[stopperm]

Description

The ClickHouse operator fails to establish secure connections to ClickHouse Keeper while direct manual Keeper connection to secure port and ClickHouse-to-Keeper connections work correctly. The error suggests a TLS protocol version mismatch or incorrect TLS configuration for operator-to-keeper connections.

Error Messages

Operator Logs:

log altinity-clickhouse-operator 2025/09/15 13:32:18 connected to 100.65.20.1:9281 altinity-clickhouse-operator 2025/09/15 13:32:18 authentication failed: read tcp 100.65.25.20:42498->100.65.20.1:9281: read: connection reset by peer altinity-clickhouse-operator 2025/09/15 13:32:18 authentication failed: read tcp 100.65.25.20:53868->100.65.29.205:9281: read: connection reset by peer

Keeper Logs:

log Code: 210. DB::NetException: SSL Exception: error:0A00010B:SSL routines::wrong version number, while reading from socket

Environment

• ClickHouse Server version: 25.6.11.18
• ClickHouse Keeper version: 25.6.11.18
• ClickHouse Operator deployment: Using Helm chart v. 0.25.3

Current Setup

1. ClickHouse Keeper Configuration:

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: keeper
spec:
  configuration:
    clusters:
      - name: "keeper-cluster"
        layout:
          replicasCount: 3
    files:
      config.xml: |        
        <clickhouse>
            <listen_host>0.0.0.0</listen_host>
            <keeper_server>
                <tcp_port_secure>9281</tcp_port_secure>
                <raft_configuration>
                  <secure>true</secure>
                  <server>
                    <id>0</id>
                    <hostname>chk-keeper-keeper-cluster-0-0.analytics.svc.cluster.local</hostname>
                    <port>9234</port>
                  </server>
                  <server>
                    <id>1</id>
                    <hostname>chk-keeper-keeper-cluster-0-1.analytics.svc.cluster.local</hostname>
                    <port>9234</port>
                  </server>
                  <server>
                    <id>2</id>
                    <hostname>chk-keeper-keeper-cluster-0-2.analytics.svc.cluster.local</hostname>
                    <port>9234</port>
                  </server>
                </raft_configuration>
            </keeper_server>
            <openSSL>
                <server>
                    <certificateFile>/etc/clickhouse-keeper/certs/tls.crt</certificateFile>
                    <privateKeyFile>/etc/clickhouse-keeper/certs/tls.key</privateKeyFile>
                    <caConfig>/etc/clickhouse-keeper/certs/selfsigned-cluster-ca-bundle.pem</caConfig>
                    <verificationMode>none</verificationMode>
                    <loadDefaultCAFile>false</loadDefaultCAFile>
                    <cacheSessions>true</cacheSessions>
                    <disableProtocols>sslv2,sslv3</disableProtocols>
                    <preferServerCiphers>true</preferServerCiphers>
                </server>
                <client>
                    <certificateFile>/etc/clickhouse-keeper/certs/tls.crt</certificateFile>
                    <privateKeyFile>/etc/clickhouse-keeper/certs/tls.key</privateKeyFile>
                    <caConfig>/etc/clickhouse-keeper/certs/selfsigned-cluster-ca-bundle.pem</caConfig>
                    <verificationMode>none</verificationMode>
                    <loadDefaultCAFile>true</loadDefaultCAFile>
                    <cacheSessions>true</cacheSessions>
                    <disableProtocols>sslv2,sslv3</disableProtocols>
                    <preferServerCiphers>true</preferServerCiphers>
                    <invalidCertificateHandler>
                        <name>RejectCertificateHandler</name>
                    </invalidCertificateHandler>
                </client>
            </openSSL>
        </clickhouse>
  defaults:
    templates:
      dataVolumeClaimTemplate: default
      podTemplate: default
  templates:
    podTemplates:
      - name: default
        spec:
          containers:
            - name: clickhouse-keeper
              volumeMounts:
                - name: clickhouse-keeper-tls-secret-volume
                  mountPath: /etc/clickhouse-keeper/certs/tls.crt
                  subPath: tls.crt
                  readOnly: true
                - name: clickhouse-keeper-tls-secret-volume
                  mountPath: /etc/clickhouse-keeper/certs/tls.key
                  subPath: tls.key
                  readOnly: true
                - name: selfsigned-cluster-ca-bundle-volume
                  mountPath: /etc/clickhouse-keeper/certs/selfsigned-cluster-ca-bundle.pem
                  subPath: selfsigned-cluster-ca-bundle.pem
                  readOnly: true
          volumes:
            - name: clickhouse-keeper-tls-secret-volume
              secret:
                secretName: clickhouse-keeper-cluster-tls-secret
            - name: selfsigned-cluster-ca-bundle-volume
              configMap:
                name: selfsigned-cluster-ca-bundle

2. ClickHouse Configuration (Working with Keeper):

    files:
      config.d/tls.xml: |
        <clickhouse>
          <!-- Disable unencrypted ports -->
          <https_port>8443</https_port>
          <tcp_port_secure>9440</tcp_port_secure>
          <listen_host>0.0.0.0</listen_host>
          <openSSL replace="replace">
            <server>
              <certificateFile>/etc/clickhouse-server/certs/tls.crt</certificateFile>
              <privateKeyFile>/etc/clickhouse-server/certs/tls.key</privateKeyFile>
              <dhParamsFile>/etc/clickhouse-server/certs/dhparam.pem</dhParamsFile>
              <verificationMode>none</verificationMode>
              <loadDefaultCAFile>false</loadDefaultCAFile>
              <caConfig>/etc/clickhouse-server/certs/selfsigned-cluster-ca-bundle.pem</caConfig> 
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
            </server>
            <client>
              <loadDefaultCAFile>true</loadDefaultCAFile>
              <caConfig>/etc/clickhouse-server/certs/selfsigned-cluster-ca-bundle.pem</caConfig>
              <certificateFile>/etc/clickhouse-server/certs/tls.crt</certificateFile>
              <privateKeyFile>/etc/clickhouse-server/certs/tls.key</privateKeyFile>
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
              <verificationMode>none</verificationMode>
              <invalidCertificateHandler>
                  <name>RejectCertificateHandler</name>
              </invalidCertificateHandler>
            </client>
          </openSSL>
        </clickhouse>
    zookeeper:
      nodes:
        - host: "chk-keeper-keeper-cluster-0-0.analytics.svc.cluster.local"
          port: 9281
          secure: "yes"
        - host: "chk-keeper-keeper-cluster-0-1.analytics.svc.cluster.local"
          port: 9281
          secure: "yes"
        - host: "chk-keeper-keeper-cluster-0-2.analytics.svc.cluster.local"
          port: 9281
          secure: "yes"``` 

**3. Operator Configuration:**

yaml configs: files: config.yaml: clickhouse: access: rootCA: | # CA certificates content

### Current Behavior
1. ClickHouse instances successfully establish TLS connections with Keeper nodes
2. The operator fails to establish secure connections with Keeper nodes
3. Keeper logs indicate a TLS protocol version mismatch
4. Connections from the operator are being reset by Keeper

### Expected Behavior
The operator should successfully establish secure TLS connections to Keeper nodes, similar to how ClickHouse instances connect.

### Questions
1. Does the operator support configuring TLS protocol versions for Keeper connections?
2. Is there a way to specify detailed TLS settings for operator-to-keeper connections?
3. What is the expected configuration for operator TLS settings when connecting to a secure Keeper?
4. Are there any known limitations or requirements for operator-to-keeper TLS connections?

### Additional Information
- All certificates are managed via cert-manager
- CA bundle is distributed via ConfigMap
- The same CA infrastructure is used for all components
- Direct connections between ClickHouse and Keeper work correctly
- TLS is required for all connections in the cluster
- The operator has access to the required certificates and CA bundle

[joscabmar]

I am bumping into the exact same issue. And it is annoying, because our organization requires TLS to be used everywhere. And this is keeping us away from adopting Clickhouse Keeper.

[stopperm]

Hi @joscabmar
same for my organization :( And I am wondering why there wasn't any issue about that before. It was very frustrating trying to find a fix without any outcome.

Do you know by any chance if zookeeper tls communications work just fine ? I guess I would be forced to use zookeeper instead then.