Fix AES-GCM nonce reuse in the async encrypt path

[embhorn]

Description

wc_AesGcmEncrypt_ex() reports the nonce it consumed via ivOut and advances an internal counter so no two records share a nonce under one key. The counter only advanced when wc_AesGcmEncrypt() returned 0; but an async backend reports a successful submission with WC_PENDING_E.

Fixes zd22291

Testing

test_wc_AesGcmEncrypt_ex_NonceUnique
test_wc_AesCcmEncrypt_ex_NonceUnique
test_tls12_aesgcm_record_nonce_unique

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

This PR fixes a critical TLS 1.2 AES-GCM security issue where the nonce/counter in wc_AesGcmEncrypt_ex() was not advanced when an asynchronous crypto backend reported a successful submission via WC_PENDING_E, allowing nonce reuse under a single traffic key. It also hardens nonce handling by using ivOut as the stable nonce buffer for backends that treat aes->reg as scratch space, and adds regression tests to detect nonce reuse in both the raw crypto API and the TLS record layer.

Changes:

• Advance the internal nonce counter on both 0 and WC_PENDING_E returns in wc_AesGcmEncrypt_ex() / wc_AesCcmEncrypt_ex(), and ensure the consumed nonce is passed/stored via ivOut rather than relying on aes->reg.
• Apply analogous nonce handling adjustments in the TI AES port’s _ex() encrypt paths.
• Add new API- and TLS-level tests that validate nonce uniqueness (and correct increment/carry behavior) across multiple encryptions/records.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
wolfcrypt/src/port/ti/ti-aes.c	Uses ivOut as the nonce source for encrypt calls and restores aes->reg after backend use to avoid scratch-space corruption.
wolfcrypt/src/aes.c	Fixes async nonce reuse by advancing the nonce counter on WC_PENDING_E and makes nonce storage stable via ivOut.
tests/api/test_tls.h	Registers the new TLS nonce-uniqueness test.
tests/api/test_tls.c	Adds a TLS 1.2 AES-GCM record-level test ensuring explicit nonces on the wire never repeat.
tests/api/test_aes.h	Registers new AES _ex() nonce-uniqueness tests.
tests/api/test_aes.c	Adds API-level tests for nonce uniqueness/increment/carry for AES-GCM/CCM _ex() encrypt paths (including async submission behavior for GCM).
doc/dox_comments/header_files/aes.h	Updates Doxygen for _ex() APIs to document nonce/IV behavior and async buffer-lifetime expectations.

Suppressed comments (1)

doc/dox_comments/header_files/aes.h:2936

• The nonce-counter behavior is documented as “advanced on every call”, but the code only advances the nonce counter when a nonce is actually consumed (success / async submission). The current wording (and the note about failure behavior) is likely to misstate what happens when the call fails before consuming a nonce.

    The nonce is taken from an internal counter that is advanced on every
    call, so no two calls under one key produce the same nonce. ivOut must not
    overlap out, in, authTag or authIn: the buffer is the working copy of the
    nonce that was consumed, and overwriting it corrupts the counter for the
    next call. ivOut may be written even when the function returns an error -

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m0plus

• FLASH: .text +92 B (+0.1%, 64,975 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m3

• FLASH: .text +164 B (+0.1%, 123,739 B / 262,144 B, total: 47% used)

gcc-arm-cortex-m4

• FLASH: .text +192 B (+0.1%, 202,031 B / 262,144 B, total: 77% used)

gcc-arm-cortex-m4-baremetal

• FLASH: .text +192 B (+0.3%, 67,619 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-crypto-only

• FLASH: .text +192 B (+0.1%, 176,170 B / 262,144 B, total: 67% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .text +192 B (+0.1%, 183,612 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-min-ecc

• FLASH: .text +128 B (+0.2%, 62,533 B / 262,144 B, total: 24% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +256 B (+0.0%, 775,868 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pkcs7

• FLASH: .text +192 B (+0.1%, 215,022 B / 262,144 B, total: 82% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +128 B (+0.0%, 297,756 B / 1,048,576 B, total: 28% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +192 B (+0.1%, 328,304 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-sp-math

• FLASH: .text +128 B (+0.2%, 62,533 B / 262,144 B, total: 24% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text +192 B (+0.2%, 124,531 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +192 B (+0.1%, 238,561 B / 262,144 B, total: 91% used)

gcc-arm-cortex-m7

• FLASH: .text +192 B (+0.1%, 202,031 B / 262,144 B, total: 77% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +192 B (+0.1%, 298,716 B / 1,048,576 B, total: 28% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +128 B (+0.1%, 238,561 B / 262,144 B, total: 91% used)

linuxkm-pie

• Data: __patchable_function_entries +16 B (+0.1%, 26,584 B)

linuxkm-standard

• Data: __patchable_function_entries -8 B (-0.0%, 49,392 B)

stm32-sim-stm32h753

• FLASH: .text +192 B (+0.1%, 186,352 B / 2,097,152 B, total: 9% used)

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11175

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src

Findings: 5
5 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.