rubygems: Add PQC ML-DSA support for cryptographically signed gems workflow

[junaruga]

This PR is related to #9542, and to add PQC ML-DSA support for cryptographically signed gems workflow.
The PR has 6 commits. The 1st - 5th commits are not related to PQC, but for preparation to implement the 6th commit, the main commit of this PR. The PR's 6th commit is in draft phase. I want to share my early phase of the implementation to get feedback and adjust the direction. I hope #9678 will be reviewed and merged before this PR.

Proof of concept

I prepared proof-of-concept scripts for signed gems workflow.

• The document is here.
• The script is here.
• The CI result is here.

Commit message

These changes enable the full PQC ML-DSA cryptographically signed
gems workflow: gem cert --build (key/cert generation), gem build
(package signing), and gem install -P HighSecurity (signature
verification).

The gem cert -A accepts ML-DSA-44, ML-DSA-65, or ML-DSA-87 to generate
ML-DSA based cert and key.

$ gem cert --build your@email.com -A ML-DSA-44
$ gem cert --build your@email.com -A ML-DSA-65
$ gem cert --build your@email.com -A ML-DSA-87

The reason why it accepts all 3 ML-DSA parameter sets rather than one of them
is because all 3 ML-DSA parameter sets suit different workflows.
While ML-DSA-65 can be used commonly, a high security environment requires
ML-DSA-87.

ML-DSA-44: NIST security strength category 2, signature size 2420 bytes
ML-DSA-65: NIST security strength category 3, signature size 3309 bytes
ML-DSA-87: NIST security strength category 5, signature size 4627 bytes

See NIST FIPS 204 Section 4 (Parameter Sets).
https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.204.pdf

See Security (Evaluation Criteria) - 4.A.5 Security Strength Categories.
https://csrc.nist.gov/projects/post-quantum-cryptography/post-quantum-cryptography-standardization/evaluation-criteria/security-(evaluation-criteria)

The workflow also includes

• gem cert -C cert.pem
• gem cert -K private_key.pem
• gem build gemname.gemspec with s.cert_chain and s.signing_key to build
  signed gem.
• gem install gemname-version.gem -P HighSecurity # or -P MediumSecurity
• gem update gemname-version.gem -P HighSecurity # or -P MediumSecurity

The changes come from:

• OpenSSL::PKey::RSA, OpenSSL::PKey::DSA, OpenSSL::PKey::EC for RSA, DSA, EC
  cases, vs OpenSSL::PKey::PKey for ML-DSA-NN cases. OpenSSL::PKey::PKey
  doesn't have the methods #to_pem, #private?.
• Ruby OpenSSL methods #sign (signing) and #verify don't accept digest
  algorithm, in ML-DSA cases because ML-DSA has a built-in digest
• Unify tool/create_certs.rb with tool/create_encrypted_key.rb
  to generate both RSA and ML-DSA-65 certificates/keys.
• ML-DSA-65 related .pem files generated by tool/create_certs.rb.
• Adding ML-DSA tests. We need lots of tests because of differences between
  OpenSSL::PKey::RSA/DSA/EC and OpenSSL::PKey::PKey mentioned above.

The method chains for the changes are below.

• gem cert --build (key/cert generation)
  • lib/rubygems/security.rb self.sign
    • chain: execute -> build -> build_cert ->
      Gem::Security.create_cert_email -> create_cert_self_signed ->
      sign
  • lib/rubygems/security.rb self.write
    • chain: execute -> build -> build_key -> Gem::Security.write
• gem cert -K (for --sign or --build)
  • lib/rubygems/commands/cert_command.rb open_private_key
    • chain: open_private_key
• gem build (package signing)
  • lib/rubygems/security/signer.rb sign
    • chain: execute -> build_gem -> build_package ->
      Gem::Package.build -> Package#build -> add_metadata ->
      TarWriter#add_file_signed -> signer.sign
• gem install -P HighSecurity (signature verification)
  • lib/rubygems/security/policy.rb check_trust
    • chain: execute -> install_gems -> install_gem ->
      RequestSet#install -> Installer#install ->
      pre_install_checks -> verify_spec -> Package#spec ->
      Package#verify -> Policy#verify_signatures ->
      Policy#verify -> check_trust
  • lib/rubygems/security/policy.rb check_data
    • chain: (same as above up to Policy#verify) -> check_data

See also https://guides.rubygems.org/security/ for the signed gems workflows.

Assisted-by: Claude:claude-opus-4-6[1m]

Notes & challenges

Differences between OpenSSL::PKey::RSA/DSA/EC and OpenSSL::PKey::PKey

OpenSSL::PKey::RSA, OpenSSL::PKey::DSA, OpenSSL::PKey::EC for RSA, DSA, EC cases, vs OpenSSL::PKey::PKey for ML-DSA-NN cases. OpenSSL::PKey::PKey doesn't have the methods #to_pem, #private?.

The above situation makes the changes complicated. Ideally I want the new feature OpenSSL::PKey::ML-DSA class or OpenSSL::PKey::ML-DSA-NN classes, or OpenSSL::PKey::PKey to implement #to_pem and #private? methods in Ruby OpenSSL. I am thinking to implement the following classes in the meantime.

• Gem::OpenSSL::PKey::ML_DSA class which is child class of OpenSSL::PKey::PKey for the compatibility with OpenSSL::PKey::RSA and etc. The class can have #to_pem and #private?.
• Gem::OpenSSL::PKey::PKey class to implement self.generate_key which is wrapper of OpenSSL::PKey.generate_key, but can return Gem::OpenSSL::PKey::ML_DSA.

After OpenSSL::PKey::ML-DSA is implemented, we can use use OpenSSL::PKey::ML-DSA if OpenSSL::PKey::ML-DSA is defined. Otherwise can use Gem::OpenSSL::PKey::ML_DSA.

or

• Gem::OpenSSL::PKey::PKey class which is child class of OpenSSL::PKey::PKey for the compatibility with OpenSSL::PKey::RSA and etc. The class can have #to_pem and #private?.
• Gem::OpenSSL::PKey::PKey class to implement self.generate_key which is wrapper of OpenSSL::PKey.generate_key, but can return Gem::OpenSSL::PKey::PKey.

Constant management in test files

test/rubygems/helper.rb defines KEY/CERT constants.

However, for example, the following files also defines such constants. I also saw duplicated constants from the ones in helper.rb.

• test/rubygems/test_gem_commands_cert_command.rb
• test/rubygems/test_gem_security.rb
• test/rubygems/test_gem_security_policy.rb
• test/rubygems/test_gem_security_signer.rb

These constants can be managed in one place.

omit_if_support_pqc

I added omit_if_support_pqc to be used in non-PQC specific tests. However, so far it is not used in any places. I will delete the method, minimizing the code changes if it is not used eventually. I used omit_if_support_pqc for an error case in non-PQC tests. However, it was hard to handle OpenSSL versions and Ruby OpenSSL versions in the error case. So, I deleted the non-PQC tests.

What was the end-user or developer problem that led to this PR?

Users cannot use ML-DSA for signed gem workflow.
https://guides.rubygems.org/security/

What is your fix for the problem, implemented in this PR?

Added the implementation to add PQC ML-DSA support for cryptographically signed gems workflow.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[junaruga]

I added "Proof of concept" section to the first comment.

[junaruga]

Differences between OpenSSL::PKey::RSA/DSA/EC and OpenSSL::PKey::PKey

For the above topic, I opened ruby/openssl#1085 for feature request.

[rhenium]

Can this be simplified like this?

However this is clearly checking the subject public key, not the certificate itself. (I haven't read the rest of the code, so I don't know what this implies.)

    raise Gem::Security::Exception,
          "trusted root certificate #{root.subject} checksum " \
          "does not match signing root certificate checksum" unless
      save_cert.public_key.to_der == root.public_key.to_der

[rhenium]

I wonder if we can just remove this and instead check for an exception that may be raised by OpenSSL::X509::Certificate#sign later.