Bugfixes for the SymCrypt provider

[mamckee]

This PR fixes a number of minor bugs and behavior issues with the SymCrypt provider. These bugs are either normally not triggerable through the EVP layer or normal calling patterns, or impact uncommonly used functions.

• Fixed an incorrect size comparison in AES when setting OSSL_CIPHER_PARAM_TLS_MAC_SIZE
• Fixed an issue where p_scossl_dh_keymgmt_dup_key_ctx would allocate an incorrectly sized buffer when copying a context using a custom group and with no key set yet.
• Added initialization state checks to HMAC to prevent usage of context with no key set, but still permit partially initialized contexts for Allow MAC context duplication for partially initialized contexts #156
• Added check for size of x25519 keys passed by parameter to the provider
• Added checks for outlen in RSA encrypt/decrypt. This was previously not checked in OpenSSL for RSA encrypt but was hardened in 3.3.7. These checks bring SymCrypt-OpenSSL in line with the default behavior.
• Fixed check in RSA key match, where the private exponent was not compared correctly.

Thanks @vcsjones for discovering these!

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[MS-megliu]

Pre-existing issue in touched code: p_scossl_rsa_cipher_dupctx() shallow-copies the SCOSSL_RSA_CIPHER_CTX struct via OPENSSL_memdup, which includes the pbLabel pointer. Both the original and duplicate contexts will call OPENSSL_free(ctx->pbLabel) on cleanup — double-free / use-after-free. Since this file is already being touched for the outlen hardening, consider deep-copying pbLabel in dupctx.

[MS-megliu]

LGTM

[MS-megliu]

LGTM