Confirmed interoperability failure on large TLS 1.3 Certificate messages

[LiD0209]

Confirmed interoperability failure on large TLS 1.3 Certificate messages

Problem Description

This candidate is a real issue, but the precise failure mode is narrower than the original static triage wording.

RFC 8446 allows TLS 1.3 CertificateEntry.cert_data and the enclosing certificate_list to use 24-bit length fields. It also explicitly allows handshake messages, including Certificate, to be fragmented across multiple TLS records. In the tested mbedTLS build, a standards-conformant TLS 1.3 peer presenting a large but valid X.509 certificate chain entry causes the handshake to fail during message intake because the implementation cannot receive the full fragmented Certificate handshake message. The parser also contains an explicit < 64 KiB cap for certificate_list and cert_data.

The practical defect is therefore:

• mbedTLS fails to interoperate with a peer that sends a valid fragmented TLS 1.3 Certificate message larger than the implementation's internal limits.

This report deduplicates multiple candidate-level records that resolved to the same root cause. It also serves as the canonical report for requirements req-1cf99135bf70a9bc95da and req-2266a43cdf9a5f5ec95a; their narrower certificate_list and cert_data parser-limit findings are components of the same large-Certificate interoperability defect.

Standard Requirement

• Official standard: RFC 8446

• Primary structure reference: RFC 8446 Appendix B, CertificateEntry and Certificate

• Additional behavior reference: RFC 8446 Section 3.4 and Section 5.1

Relevant RFC text:

opaque cert_data<1..2^24-1>;

CertificateEntry certificate_list<0..2^24-1>;

RFC 8446 also defines variable-length vectors as legal ranges with inclusive bounds, and states that handshake messages may be fragmented across multiple records. For Certificate, this matters because the message can be much larger than a single TLS record while still remaining conformant.

Interpretation:

• For TLS 1.3 with X.509 certificates, cert_data is encoded with a 24-bit length field and is syntactically legal up to 2^24 - 1 octets.
• The enclosing certificate_list is also encoded with a 24-bit length field and is syntactically legal up to 2^24 - 1 octets.
• TLS 1.3 handshake messages may be fragmented across records, so a conformant implementation must not assume that a legal Certificate handshake message fits into a single 16 KiB content buffer.

Relevant Source Code

1. TLS 1.3 certificate parser imposes a < 64 KiB cap

File: library/ssl_tls13_generic.c

    /* In theory, the certificate list can be up to 2^24 Bytes, but we don't
     * support anything beyond 2^16 = 64K.
     */
    if ((certificate_request_context_len != 0) ||
        (certificate_list_len >= 0x10000)) {
        MBEDTLS_SSL_DEBUG_MSG(1, ("bad certificate message"));
        MBEDTLS_SSL_PEND_FATAL_ALERT(MBEDTLS_SSL_ALERT_MSG_DECODE_ERROR,
                                     MBEDTLS_ERR_SSL_DECODE_ERROR);
        return MBEDTLS_ERR_SSL_DECODE_ERROR;
    }

Later in the same function:

        /* In theory, the CRT can be up to 2^24 Bytes, but we don't support
         * anything beyond 2^16 = 64K. Otherwise as in the TLS 1.2 code,
         * check that we have a minimum of 128 bytes of data, this is not
         * clear why we need that though.
         */
        if ((cert_data_len < 128) || (cert_data_len >= 0x10000)) {
            MBEDTLS_SSL_DEBUG_MSG(1, ("bad Certificate message"));
            MBEDTLS_SSL_PEND_FATAL_ALERT(MBEDTLS_SSL_ALERT_MSG_DECODE_ERROR,
                                         MBEDTLS_ERR_SSL_DECODE_ERROR);
            return MBEDTLS_ERR_SSL_DECODE_ERROR;
        }

These checks show an explicit implementation limit below the RFC's 24-bit syntax range.

2. Default input content buffer is 16384

File: include/mbedtls/ssl.h

#if !defined(MBEDTLS_SSL_IN_CONTENT_LEN)
#define MBEDTLS_SSL_IN_CONTENT_LEN 16384
#endif

3. Fragmented handshake intake fails when the next read would exceed the input buffer

File: library/ssl_msg.c

    if (nb_want > in_buf_len - (size_t) (ssl->in_hdr - ssl->in_buf)) {
        MBEDTLS_SSL_DEBUG_MSG(1, ("requesting more data than fits"));
        return MBEDTLS_ERR_SSL_BAD_INPUT_DATA;
    }

Observed behavior and code together show two stacked limits:

• the default build cannot receive arbitrarily large fragmented handshake messages because of the fixed input buffer;
• even if the handshake buffering issue were bypassed, the TLS 1.3 certificate parser contains an explicit < 64 KiB limit for certificate_list and each cert_data.

Runtime Evidence

Environment

• Date of rerun: 2026-08-04
  The large test certificate was a valid self-signed ECDSA P-256 certificate whose DER length was 70317 bytes, which is above 65535 but well below the RFC's 2^24 - 1 structural limit.

Positive control: another TLS 1.3 implementation accepts the same server certificate

I started a local Python TLS 1.3 server with the generated large ECDSA
certificate and connected to it using Python's TLS client stack.

Observed client output:

client_ok TLSv1.3 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)

The server reported:

listening
accepted ('127.0.0.1', 31007)
handshake_ok TLSv1.3

This confirms that the server certificate used for the reproduction is acceptable to another TLS 1.3 implementation and that the test setup itself is sound.

Reproducer: mbedTLS client fails against the same TLS 1.3 server

I then connected the mbedTLS ssl_client2.exe sample to the same local server
with:

server_addr=127.0.0.1
server_port=4443
force_version=tls13
auth_mode=none
debug_level=2

The mbedTLS client reported:

ssl_msg.c:1873: |1| requesting more data than fits
! mbedtls_ssl_handshake returned -0x87

The same log also shows that the peer had already started delivering a large TLS 1.3 Certificate handshake message:

ssl_msg.c:2005: |2| in_left: 5, nb_want: 16406
ssl_msg.c:2028: |2| ssl->f_recv(_timeout)() returned 16401
...
ssl_msg.c:1873: |1| requesting more data than fits

From the earlier detailed run of the same reproducer, the client had already recognized the incoming message as a fragmented TLS 1.3 Certificate with total handshake length 70330 bytes before failing on the next fetch.

After the mbedTLS abort, the server reported:

listening
accepted ('127.0.0.1', 24211)
handshake_error ConnectionResetError(...)

Interpretation:

• the peer sent a valid TLS 1.3 handshake sequence;
• the message was large and fragmented, which RFC 8446 allows;
• mbedTLS aborted because the next handshake read would exceed its input buffer.

Inconsistency Reason

• RFC 8446 permits large CertificateEntry.cert_data and certificate_list values encoded with 24-bit lengths and permits fragmented handshake delivery.
• The tested mbedTLS build fails to receive a valid fragmented Certificate handshake message carrying a 70317-byte DER certificate.
• The TLS 1.3 parser additionally contains explicit < 64 KiB checks for both certificate_list_len and cert_data_len.

Decision Reason

The issue is confirmed by actual execution, not just static review.

• The RFC syntax and fragmentation rules allow the tested message shape.
• A separate TLS 1.3 implementation completed the handshake with the same server certificate, which rules out the certificate being malformed or the local test harness being fundamentally broken.
• mbedTLS failed on the same peer because of implementation-specific size handling during TLS 1.3 Certificate processing.

Therefore the correct verdict is issue_found.

Scope Notes

• The strongest runtime evidence here is for large, standards-conformant fragmented Certificate messages causing interoperability failure in the default build.
• The cert_data_len < 128 branch in ssl_tls13_generic.c still exists, but this rerun did not find a practical X.509 certificate below that size. It should not be treated as the primary reproduced defect.
• The parser's explicit < 64 KiB checks remain relevant because they are stricter than the RFC structure even aside from the earlier handshake-buffer failure observed in the default build.

Consolidation Record

Absorbed requirement IDs: req-1cf99135bf70a9bc95da and req-2266a43cdf9a5f5ec95a. Their 24-bit vector-bound and fragmented-message evidence remains represented by the parser checks and large-certificate runtime evidence in this canonical report.