crypto: Add support for sha512

[Steven Malis (smalis-msft)]

And move one bit from windows into hashes to keep things as contained as we can.

[github-actions]

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

[Copilot]

Pull request overview

Adds SHA-512 as a supported hash algorithm in the support/crypto crate and wires it through RSA operations across backends, while also centralizing Windows RSA signature-OID parsing into hashes.rs to reduce Windows-specific logic in the X.509 backend.

Changes:

• Extend HashAlgorithm with Sha512 and add backend-specific mappings (SymCrypt/OpenSSL/BCrypt/macOS Security.framework, plus DER OID support).
• Add SHA-512 support to the RustCrypto RSA backend for OAEP, PKCS#1 v1.5, and PSS.
• Refactor Windows X.509 verification to use a shared RSA signature-OID → HashAlgorithm helper and expand OAEP round-trip coverage to include SHA-512.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
support/crypto/src/x509/win.rs	Refactors signature-algorithm OID parsing to use a shared helper, keeping Windows X.509 code smaller.
support/crypto/src/rsa/rust.rs	Implements RSA OAEP/PKCS#1/PSS SHA-512 support for the RustCrypto backend.
support/crypto/src/rsa/mod.rs	Expands OAEP round-trip tests to cover all supported hash algorithms, including SHA-512.
support/crypto/src/hashes.rs	Adds Sha512 to HashAlgorithm, backend mappings, DER OID mapping, and the Windows RSA signature-OID helper.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

2 Petri tests failed (0 unstable)

[Ben Hillis (benhillis)]

Steven Malis (@smalis-msft), please either backport this change to release/1.8.2607 or remove the backport_1.8.2607 label. Thank you!