constant_time: fix the -Os inlining loss, and add an Xtensa assembly path

[sweetlilmre]

Intro

While building a firmware for an esp32 device and migrating from Arduino / IDF to pure IDF I noticed a failure during TLS negotiation. This was due to the TLS handshake taking longer than the watchdog timeout and causing a reboot during mid fetch. In investigating this behaviour, I found a regression in performance due to constant time hardening.

AI was used in both the development and analysis of this patch, specifically Claude Opus 5 on Medium level thinking. The patch was adversarially reviewed by myself through several iterations, constant-time testing was done exposing a coverage gap in the current code and purpose test harnesses built to validate the approach.

Summary

Two independent problems make the constant-time primitives expensive on embedded targets, both on the bignum hot path since mbedtls_mpi_core_sub() and mbedtls_mpi_core_mla() were made constant time (mbedtls 4b4869a15, TF-PSA-Crypto 77bd4798). On an ESP32 this costs 15-41% per bignum primitive against mbedTLS 3.6.6, at identical call counts.

1. At -Os these functions are not inlined, so every use pays a call. This affects targets that already have an assembly path, including Arm.
2. Xtensa has no assembly path, so it uses the generic C built from mbedtls_ct_compiler_opaque() barriers.

The third commit is an unrelated small cleanup at two call sites, separated deliberately so it can be dropped without affecting the rest.

Why -Os matters here

mbedtls_mpi_core_sub() calls mbedtls_ct_uint_lt() twice per limb. Whether that call survives:

toolchain	-Os	-O2
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4	2 calls, not inlined	0 calls
arm-none-eabi-gcc -marm -mcpu=arm7tdmi	2 calls, not inlined	0 calls
xtensa-esp-elf-gcc	2 calls, not inlined	—
x86-64 gcc and clang	0 calls	0 calls

x86-64 inlines regardless, and does so for the generic C body too, so the difference is the target rather than the code. That is presumably why this has gone unnoticed. Arm Cortex-M -Os builds pay it today.

MBEDTLS_CT_INLINE is applied only where an assembly path is in use. Cost on Arm thumb -Os, whole translation unit: +8 bytes of .text. Applying it to the generic C fallback instead costs +396 bytes, which is a size/speed trade-off rather than a clear win, so that has been left alone.

Measurements

ESP32-D0WD-V3 (Xtensa LX6) and ESP32-S3 (Xtensa LX7), both at 240 MHz, GCC 14.2, -Os, RSA/MPI accelerator off so this is pure library C. Offline benchmark: no WiFi, no TLS, fixed operands, one task pinned to a core, batches sized to ~200 ms, min/mean/max over 5 batches. Reference is mbedTLS 3.6.6 from ESP-IDF v5.5.5 on the same device.

ESP32 (LX6):

bench	3.6.6	4.1.0	gap	with this series	vs 3.6.6
ecdsa_verify P-256	245.12 ms	327.75 ms	+33.7%	243.19 ms	−0.8%
ecp_mul P-256	113.92 ms	151.56 ms	+33.0%	113.99 ms	+0.1%
mpi_inv_mod P-256	8.66 ms	12.23 ms	+41.2%	7.59 ms	−12.3%
mpi_mul 256-bit	9.78 µs	11.28 µs	+15.4%	9.80 µs	+0.2%

ESP32-S3 (LX7):

bench	3.6.6	4.1.0	gap	with this series	vs 3.6.6
ecdsa_verify P-256	189.38 ms	266.97 ms	+41.0%	193.97 ms	+2.4%
ecp_mul P-256	87.40 ms	122.22 ms	+39.8%	89.57 ms	+2.5%
mpi_inv_mod P-256	7.40 ms	10.68 ms	+44.3%	6.82 ms	−7.9%

The regression is worse on the newer core, and the series recovers 94% of it there against 100% on LX6.

Attribution is exact: reverting only the constant-time lines in mbedtls_mpi_core_sub() and mbedtls_mpi_core_mla() to the 3.6 plain-C form recovers the whole gap on its own, which is what pins the cause to them rather than to anything else that changed between 3.6 and 4.x.

The regression is O(n) at roughly 44 cycles per limb, flat from 256 to 4096 bits. mbedtls_mpi_core_mul() calls mla once per limb of B with excess_len == 1, so an n×n multiply runs the changed line exactly n times, and each mbedtls_mpi_core_montmul() performs 6n such operations. Because the multiply itself is O(n²) the relative cost shrinks as operands grow (+15.4% at 256 bits, +2.0% at 4096), so it is worst exactly where ECC lives.

End to end, in real firmware

The tables above are primitives. The effect on a real TLS 1.2 ECDHE-ECDSA handshake to api.github.com, from the ESP32 firmware, timed from the host, six reps each:

	min	mean
unpatched 4.1.0	4089 ms	4198 ms
with this series	3286 ms	3391 ms
unpatched again	4045 ms	4253 ms

About 830 ms, or 19.8%. It was run as A-B-A because a single before/after against a live third-party host proves nothing: the two unpatched runs bracket the patched one and agree with each other to within 55 ms of mean. The firmware is also 144 bytes smaller.

That 19.8% is conditional, and in one configuration the series is a regression. It is measured with the ESP32 RSA/MPI accelerator disabled (MBEDTLS_HARDWARE_MPI off), which is this firmware's shipped setting for reasons unrelated to this patch. With the accelerator enabled, the same series makes the same handshake roughly 875 ms slower, measured as A-B-A on one device in one session, four reps each, timed inside the firmware:

	TLS phase	total
unpatched	5895 ms	6211 ms
with this series	6823 ms	7097 ms
unpatched again	5948 ms	6224 ms

The two unpatched runs bracket the patched one and agree with each other to within 13 ms of total, so this is not drift.

The cause is not established. The plausible mechanism is code growth from the forced inlining costing flash-cache misses on a part that executes from cached external flash — which would be consistent with an earlier finding on this same chip that building mbedTLS at -O2 also made TLS slower rather than faster — but that is a hypothesis and has not been measured. Note also that with the accelerator on, ESP-IDF substitutes its own mbedtls_mpi_mul_mpi, so the bignum hot path is largely off the code this series touches, and there is less for it to win back.

So the honest summary is: a clear win where the constant-time helpers are actually on the hot path, and a measurable loss on ESP32 builds that use the RSA/MPI accelerator. If that trade is unacceptable, patch 1/3 (the forced inlining) is the piece to reconsider — 2/3 and 3/3 do not grow code the same way.

Constant-time testing, including a rejected approach

A pure-C fix was tried first: rewriting the generic mbedtls_ct_uint_lt() as the same branchless six-operation sequence the Arm path uses, with no barriers. It is faster, and it is not constant time — clang 22 at -O2 recognises the idiom, re-derives x < y and emits a conditional branch. Keeping barriers on only the two inputs does not help either, because a barrier hides a value and not the structure of an expression.

Both were caught with MBEDTLS_TEST_CONSTANT_FLOW_MEMSAN and a negative control:

variant	result
plain-C carry, mbedTLS 3.6 style	FAIL — the negative control, showing the harness detects the property
unmodified upstream 4.1.0	PASS
branchless C, no barriers	FAIL
branchless C, barriers on inputs only	FAIL
the _if_else_0 change in commit 3	PASS

That is what led to assembly, which is opaque to the optimiser by construction. It also says the existing design is right and the assembly paths are load-bearing.

A gap in existing constant-flow coverage

Investigating the above surfaced a second issue: test_suite_bignum_core's mpi_core_sub and mpi_core_mla cases cannot validate the generic C path as they stand. They compare the returned carry (test_suite_bignum_core.function:825) while the inputs are still TEST_CF_SECRET, and the carry is secret-derived by construction, so under MemSan they report for any implementation including unmodified upstream. This was verified: stock and a plain-C carry produce byte-identical reports.

On x86 builds with MBEDTLS_HAVE_ASM the inline assembly launders the MemSan poison and the cases pass. The consequence is that the generic C constant-time path appears to have no constant-flow coverage at present, since it is only ever exercised on architectures that do not use it. That seems worth addressing independently of this PR.

What has not been verified

• The Xtensa assembly is not MemSan-tested, because there is no MemSan for Xtensa. It rests on the same argument as the existing Arm and x86 paths, plus disassembly: with this series mbedtls_mpi_core_sub() contains one conditional branch, the loop back-edge on the public limb count, identical to stock, and no calls at all. The project's own constant-flow suite should be run before accepting.
• No Arm timing. Only codegen and .text size were measured on Arm; the timings above are Xtensa.
• ESP32-S2 builds clean with codegen checked, but no S2 hardware was available to run it.
• The RISC-V Espressif parts (C2, C3, C5, C6, H2, P4) are not covered and keep the full regression. They have no assembly path either, so commit 1 is a no-op for them. A RISC-V path would be separate work.
• Everything here is GCC and clang at -Os/-O2; no other compilers or optimisation levels were tested.
• The ~875 ms regression on ESP32 builds with the RSA/MPI accelerator enabled has a measured magnitude but no established cause. The flash-cache hypothesis above is untested.

The commits

1. constant_time: force inlining where an assembly path is in use — independent, and the one that helps existing Arm users.
2. constant_time: add an Xtensa assembly path — additive #elif branches; cannot affect an architecture that already has a path. Applies on top of 1, which it extends by one line.
3. bignum_core: use _if_else_0 where the zero branch is unused — independent and architecture-neutral. Worth about 1.2% in the one configuration where it was isolated, so if it is contentious it should be dropped rather than hold up the other two.

These can be split into separate PRs, commit 3 dropped, or a RISC-V path added, if any of that would help. The harness and the raw captures behind every number above can be shared if useful.