bug: mTLS stream data to external server, APISIX not sending client certificate

[bharanitharanm]

Current Behavior

Issue: mTLS is enabled on the external server. APISIX is configured as a client to send the data along with client certificate, But the external server is reporting client certificate is not part of the request

Expected Behavior

APISIX sends the configured client certificate to external server which has mTLS enabled (TLS Verify ON)

Error Logs

Error in TLS server,

ERROR
40A764BDA57F0000:error:0A0000C7:SSL routines:tls_process_client_certificate:peer did not return a certificate:../ssl/statem/statem_srvr.c:3509:
shutting down SSL
CONNECTION CLOSED

Error in APISIX

2025/07/29 10:10:16 [crit] 51#51: *4363 SSL_read() failed (SSL: error:0A00045C:SSL routines::tlsv13 alert certificate required:SSL alert number 116) while proxying and reading from upstream, client: 100.64.64.1, server: 0.0.0.0:9100, upstream: "10.12.135.120:4433", bytes from/to client:10/0, bytes from/to upstream:0/10

Steps to Reproduce

• Enabled stream on port 9100

  stream:
    enabled: true
    tcp:
    - addr: 9100 # Port for TCP proxy to tcp-server

• Configure stream route for mTLS using below API

curl http://127.0.0.1:9180/apisix/admin/stream_routes/1   -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1"   -X PUT   -d '{
    "server_port": 9100,
    "upstream": {
      "scheme": "tls",
      "type": "roundrobin",
      "nodes": {
        "10.12.135.120:4433": 1
      },
      "tls": {
        "client_cert": "'"$(cat client.cer)"'",
        "client_key": "'"$(cat client.key)"'",
        "ca_cert": "'"$(cat ca.cer)"'",
        "verify": true
      }
    }
  }'

Start the TLS server,

openssl s_server -accept 4433 -cert server.cer -key server.key -CAfile ca.cer -Verify 1

Send data,

echo "mTLS test" | nc 10.177.193.251 9100

APISIX version - 3.11

Environment

• APISIX version (run apisix version): 3.11
• Operating system (run uname -a): Helm Charts deployed in K3S
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.25.3.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 10.1.0
• APISIX Dashboard version, if relevant: NA
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[bharanitharanm]

Any update on this issue? Is this feature available in APISIX

[Baoyuantop]

Hi @bharanitharanm, thanks for your report, we will check it when we have time.

[SkyeYoung]

@bharanitharanm Hello, after checking the code and reproducing the issue, the situation you described does indeed exist.

This is because the code below does not handle client certificates in the stream subsystem:

apisix/apisix/upstream.lua

Lines 263 to 264 in 09f6e36

	local scheme = up_conf.scheme
	if (scheme == "https" or scheme == "grpcs") and up_conf.tls then

But may I ask what your scenario is? We look forward to your feedback.

[bharanitharanm]

@SkyeYoung , thank you for confirming the issue and identifying the root cause.

Usecase - We have a Kubernetes cluster where APISIX acts as a stream proxy to TCP traffic to external services that require mutual TLS authentication.

1. Client → APISIX (stream proxy on port 9100) → External mTLS-enabled server
2. The external server requires client certificate verification for security compliance
3. APISIX needs to present a client certificate when establishing the TLS connection to the upstream

[krishnajinka]

HI, Can we know if its possible to fix this issue, or plan to fix this issue in the upstream ?
We want to try and adopt this API gateway for our product, however due to this mTLS issue we are unable to,
Hence trying to understand if you think this issue needs to be fixed or could be fixed in near future

[krishnajinka]

Hello @Baoyuantop ,

Thank you for adding this into ROADMAP. We are very much looking for adopting the APISIX as an api gateway in our product., KIndly provide a tentative date for this bug fix, or being available in any of the nearest releases
CC: @bharanitharanm
Thanks,
Krishna