RFC9846 mbedTLS issue: AES-GCM early-data key-usage limit is not enforced
Summary
The issue is real, but it should be stated narrowly. mbedTLS enforces the TLS 1.3 early-data byte quota through max_early_data_size, but that quota only counts Application Data payload bytes. It does not enforce the separate RFC9846 Section 5.5 cryptographic key-usage limit for AES-GCM early-data records.
This matters for zero-length early-data writes. A zero-length Application Data write is still encoded as a TLS 1.3 TLSInnerPlaintext, encrypted under the early-data traffic key, and assigned a new outbound record sequence number. In mbedTLS, that write returns 0, so total_early_data_size remains unchanged while the AES-GCM record/key usage still advances.
Standard Requirement
Official standard references:
- RFC 9846, Section 4.7.1, New Session Ticket Message:
max_early_data_size defines the maximum 0-RTT data size, and only Application Data payload bytes are counted.
- RFC 9846, Section 5.4, Record Padding: Application Data records may contain a zero-length
TLSInnerPlaintext.content.
- RFC 9846, Section 5.5, Limits on Key Usage: cryptographic limits apply to plaintext encrypted under a given set of keys; implementations must close the connection or perform KeyUpdate before reaching those limits; because early data cannot use KeyUpdate, senders
MUST NOT exceed the limits when sending early data. The same section gives the AES-GCM per-key record/plaintext safety bound.
Short normative excerpts:
Only Application Data payload ... is counted.
implementations MUST NOT exceed the limits when sending early data.
Interpretation:
max_early_data_size and the Section 5.5 AEAD/key-usage limit are different requirements. The early-data byte quota can be measured in Application Data payload bytes, but the cryptographic limit must account for encrypted records/plaintext under the early-data traffic key. Because KeyUpdate is unavailable for early data, the sender's only compliant choices before reaching the AEAD limit are to stop sending early data or close/progress the connection so the early-data key is no longer used.
For AES-GCM, the RFC references the AEAD-LIMITS analysis. The focused runtime probe uses 2^40 one-block zero-content records as the injected boundary state, which is deliberately above the derived short-record AES-GCM safety bounds and still far below the TLS 64-bit sequence-number wrap limit.
Relevant Source Code
The early-data write path limits only payload bytes:
/* library/ssl_msg.c:6022 */
remaining = ssl->session_negotiate->max_early_data_size -
ssl->total_early_data_size;
/* library/ssl_msg.c:6046 */
ret = ssl_write_real(ssl, buf, len);
if (ret >= 0) {
ssl->total_early_data_size += ret;
}ssl_write_real() explicitly permits len == 0, writes an Application Data record, and returns len:
/* library/ssl_msg.c:5858 */
/* Therefore, it is possible that the input message length is 0 ... */
/* library/ssl_msg.c:5903 */
ssl->out_msglen = len;
ssl->out_msgtype = MBEDTLS_SSL_MSG_APPLICATION_DATA;
/* library/ssl_msg.c:5909 */
ret = mbedtls_ssl_write_record(ssl, SSL_FORCE_FLUSH);
/* library/ssl_msg.c:5915 */
return (int) len;
TLS 1.3 then wraps and encrypts the record:
/* library/ssl_msg.c:853 */
if (transform->tls_version == MBEDTLS_SSL_VERSION_TLS1_3) {
size_t padding = ssl_compute_padding_length(rec->data_len,
MBEDTLS_SSL_CID_TLS1_3_PADDING_GRANULARITY);
ssl_build_inner_plaintext(data, &rec->data_len, post_avail, rec->type, padding);
rec->type = MBEDTLS_SSL_MSG_APPLICATION_DATA;
}
/* library/ssl_msg.c:1043 */
status = psa_aead_encrypt(transform->psa_key_enc,
transform->psa_alg,
iv, transform->ivlen,
add_data, add_data_len,
data, rec->data_len,
data, rec->buf_len - (data - rec->buf),
&rec->data_len);The implemented outbound limit is only generic 64-bit sequence wrapping:
/* library/ssl_msg.c:2749 */
for (i = 8; i > mbedtls_ssl_ep_len(ssl); i--) {
if (++ssl->cur_out_ctr[i - 1] != 0) {
break;
}
}
/* library/ssl_msg.c:2755 */
if (i == mbedtls_ssl_ep_len(ssl)) {
return MBEDTLS_ERR_SSL_COUNTER_WRAPPING;
}No AES-GCM early-data key-usage threshold was found on this path.
Implementation Behavior
For nonzero early-data payloads, total_early_data_size increases by the number of application bytes returned from ssl_write_real(). For zero-length early-data payloads, ssl_write_real() returns 0, so the byte quota never decreases.
However, the write still produces a TLS 1.3 Application Data record. The record layer adds the inner content type and padding, encrypts the resulting TLSInnerPlaintext with psa_aead_encrypt(), and increments cur_out_ctr. As a result, zero-length early-data writes consume AES-GCM early-data key usage while bypassing the only early-data quota tracked by mbedTLS.
The positive control in the runtime probe shows that mbedTLS does reject the generic 64-bit sequence wrap condition. The missing check is the earlier AES-GCM cryptographic key-usage limit required by RFC9846 Section 5.5 for early data.
Inconsistency Reason
RFC9846 requires sending implementations to stay below cryptographic key-usage limits, and it specifically notes that early data cannot use KeyUpdate. mbedTLS only enforces max_early_data_size based on Application Data payload bytes and the generic 64-bit sequence wrap limit.
Those checks are insufficient for AES-GCM early data. A zero-length Application Data write has zero counted payload bytes, but it still creates an encrypted TLS 1.3 record under the early-data traffic key. Therefore an application can repeatedly advance AES-GCM usage without advancing total_early_data_size, and mbedTLS will continue accepting writes even after an injected state beyond the AES-GCM short-record usage boundary.
Runtime Evidence
The focused AES-GCM early-data boundary probe was rebuilt and rerun from the workspace root on 2026-08-10. The run forced TLS 1.3 with TLS1-3-AES-128-GCM-SHA256, accepted a ticket that allowed early data, moved the client into the early-data write state, repeatedly sent zero-length early-data writes, injected an AES-GCM usage boundary state at 2^40, and then performed a positive control at the generic 64-bit TLS sequence wrap boundary.
Result:
- Exit code:
0
- Forced ciphersuite:
TLS1-3-AES-128-GCM-SHA256
- Ticket accepted early data:
ticket_ret=0, ticket_max_early=1024
- Client reached early-data writable state:
move_to_early_write_ret=0, early_state=3
- 1000 zero-length early-data writes succeeded while
total_early=0.
- After injecting
cur_out_ctr = 2^40, another zero-length early-data write still returned 0.
- Positive control at
UINT64_MAX returned MBEDTLS_ERR_SSL_COUNTER_WRAPPING.
Representative log:
ticket_ret=0 ticket_ciphersuite_id=0x1301 ticket_ciphersuite=TLS1-3-AES-128-GCM-SHA256 ticket_max_early=1024
client_init_ret=0
server_init_ret=0
socket_connect_ret=0
set_session_ret=0
move_to_early_write_ret=0 early_state=3 total_early=0 out_ctr=0 buffered=322
aesgcm_repeat_zero_writes_ok=1000 last_ret=0 total_early=0 out_ctr=1000 buffered=37328
aesgcm_over_limit_zero_write_ret=0 total_early=0 out_ctr_before=1099511627776 out_ctr_after=1099511627777 buffered_delta=37 early_state=3
sequence_wrap_control_ret=-27520 expected_ret=-27520 total_early=0 out_ctr_before=18446744073709551615 out_ctr_after=0 buffered_delta=0 early_state=3
Runtime interpretation:
The runtime result confirms the exact bypass condition. max_early_data_size remains unconsumed because the application payload length is zero, but each write still advances the outbound TLS record sequence. The over-limit injection demonstrates that the implementation does not reject AES-GCM early-data writes at a cryptographic usage boundary below 64-bit sequence wrap.
Impact
This is a standards compliance and cryptographic safety-margin issue rather than a small-input memory safety bug. In normal deployments, reaching the AES-GCM limit through early data is expensive and unusual, especially because nonzero payloads are also constrained by max_early_data_size. The vulnerable shape is padding-only or zero-content early-data traffic under AES-GCM, where record/key usage can grow without consuming the configured early-data byte quota.
The practical severity is therefore limited, but the RFC9846 requirement is explicit: an implementation sending early data must not exceed the cryptographic key-usage limits.
Fix Direction
Track early-data AEAD/key usage separately from total_early_data_size. For AES-GCM TLS 1.3 early data, stop accepting new early-data writes before the relevant Section 5.5 safety boundary is reached, including zero-length and padding-only records.
The check should run on the send path before mbedtls_ssl_write_record() consumes another early-data record number. It should be ciphersuite-aware: ChaCha20-Poly1305 can continue to rely on sequence-number wrap as the tighter bound described by RFC9846, while AES-GCM needs an earlier threshold.
RFC9846 mbedTLS issue: AES-GCM early-data key-usage limit is not enforced
Summary
The issue is real, but it should be stated narrowly. mbedTLS enforces the TLS 1.3 early-data byte quota through
max_early_data_size, but that quota only counts Application Data payload bytes. It does not enforce the separate RFC9846 Section 5.5 cryptographic key-usage limit for AES-GCM early-data records.This matters for zero-length early-data writes. A zero-length Application Data write is still encoded as a TLS 1.3
TLSInnerPlaintext, encrypted under the early-data traffic key, and assigned a new outbound record sequence number. In mbedTLS, that write returns0, sototal_early_data_sizeremains unchanged while the AES-GCM record/key usage still advances.Standard Requirement
Official standard references:
max_early_data_sizedefines the maximum 0-RTT data size, and only Application Data payload bytes are counted.TLSInnerPlaintext.content.MUST NOTexceed the limits when sending early data. The same section gives the AES-GCM per-key record/plaintext safety bound.Short normative excerpts:
Interpretation:
max_early_data_sizeand the Section 5.5 AEAD/key-usage limit are different requirements. The early-data byte quota can be measured in Application Data payload bytes, but the cryptographic limit must account for encrypted records/plaintext under the early-data traffic key. Because KeyUpdate is unavailable for early data, the sender's only compliant choices before reaching the AEAD limit are to stop sending early data or close/progress the connection so the early-data key is no longer used.For AES-GCM, the RFC references the AEAD-LIMITS analysis. The focused runtime probe uses
2^40one-block zero-content records as the injected boundary state, which is deliberately above the derived short-record AES-GCM safety bounds and still far below the TLS 64-bit sequence-number wrap limit.Relevant Source Code
The early-data write path limits only payload bytes:
ssl_write_real()explicitly permitslen == 0, writes an Application Data record, and returnslen:TLS 1.3 then wraps and encrypts the record:
The implemented outbound limit is only generic 64-bit sequence wrapping:
No AES-GCM early-data key-usage threshold was found on this path.
Implementation Behavior
For nonzero early-data payloads,
total_early_data_sizeincreases by the number of application bytes returned fromssl_write_real(). For zero-length early-data payloads,ssl_write_real()returns0, so the byte quota never decreases.However, the write still produces a TLS 1.3 Application Data record. The record layer adds the inner content type and padding, encrypts the resulting
TLSInnerPlaintextwithpsa_aead_encrypt(), and incrementscur_out_ctr. As a result, zero-length early-data writes consume AES-GCM early-data key usage while bypassing the only early-data quota tracked by mbedTLS.The positive control in the runtime probe shows that mbedTLS does reject the generic 64-bit sequence wrap condition. The missing check is the earlier AES-GCM cryptographic key-usage limit required by RFC9846 Section 5.5 for early data.
Inconsistency Reason
RFC9846 requires sending implementations to stay below cryptographic key-usage limits, and it specifically notes that early data cannot use KeyUpdate. mbedTLS only enforces
max_early_data_sizebased on Application Data payload bytes and the generic 64-bit sequence wrap limit.Those checks are insufficient for AES-GCM early data. A zero-length Application Data write has zero counted payload bytes, but it still creates an encrypted TLS 1.3 record under the early-data traffic key. Therefore an application can repeatedly advance AES-GCM usage without advancing
total_early_data_size, and mbedTLS will continue accepting writes even after an injected state beyond the AES-GCM short-record usage boundary.Runtime Evidence
The focused AES-GCM early-data boundary probe was rebuilt and rerun from the workspace root on 2026-08-10. The run forced TLS 1.3 with
TLS1-3-AES-128-GCM-SHA256, accepted a ticket that allowed early data, moved the client into the early-data write state, repeatedly sent zero-length early-data writes, injected an AES-GCM usage boundary state at2^40, and then performed a positive control at the generic 64-bit TLS sequence wrap boundary.Result:
0TLS1-3-AES-128-GCM-SHA256ticket_ret=0,ticket_max_early=1024move_to_early_write_ret=0,early_state=3total_early=0.cur_out_ctr = 2^40, another zero-length early-data write still returned0.UINT64_MAXreturnedMBEDTLS_ERR_SSL_COUNTER_WRAPPING.Representative log:
Runtime interpretation:
The runtime result confirms the exact bypass condition.
max_early_data_sizeremains unconsumed because the application payload length is zero, but each write still advances the outbound TLS record sequence. The over-limit injection demonstrates that the implementation does not reject AES-GCM early-data writes at a cryptographic usage boundary below 64-bit sequence wrap.Impact
This is a standards compliance and cryptographic safety-margin issue rather than a small-input memory safety bug. In normal deployments, reaching the AES-GCM limit through early data is expensive and unusual, especially because nonzero payloads are also constrained by
max_early_data_size. The vulnerable shape is padding-only or zero-content early-data traffic under AES-GCM, where record/key usage can grow without consuming the configured early-data byte quota.The practical severity is therefore limited, but the RFC9846 requirement is explicit: an implementation sending early data must not exceed the cryptographic key-usage limits.
Fix Direction
Track early-data AEAD/key usage separately from
total_early_data_size. For AES-GCM TLS 1.3 early data, stop accepting new early-data writes before the relevant Section 5.5 safety boundary is reached, including zero-length and padding-only records.The check should run on the send path before
mbedtls_ssl_write_record()consumes another early-data record number. It should be ciphersuite-aware: ChaCha20-Poly1305 can continue to rely on sequence-number wrap as the tighter bound described by RFC9846, while AES-GCM needs an earlier threshold.