Feature request: OpenSSL::PKey::PKey, OpenSSL::PKey::ML-DSA, or OpenSSL::PKey::ML-DSA-NN to implement #private? and #to_pem

[junaruga]

For the last commit of ruby/rubygems#9697: rubygems: Add PQC ML-DSA support for cryptographically signed gems workflow, there are some code changes that come from the differences between OpenSSL::PKey::RSA, OpenSSL::PKey::DSA, OpenSSL::PKey::EC classes (RSA, DSA, EC cases) that have #private? and #to_pem, and OpenSSL::PKey::PKey class (ML-DSA cases) that doesn't have #private? and #to_pem.

I guess OpenSSL::PKey::PKey class is for general algorithm. I want OpenSSL::PKey::PKey class to implement missing methods for the compatibility of OpenSSL::PKey::RSA, OpenSSL::PKey::DSA, OpenSSL::PKey::EC classes. Or I want OpenSSL::PKey::ML-DSA or OpenSSL::PKey::ML-DSA-NN classes to have the missing methods. I understand that the usage #to_pem is not encouraged due to #public_to_pem or #private_to_pem used in this case instead of #to_pem. However, I still want the #to_pem method for the compatibility.

With this feature, I can implement new ML-DSA feature without conditional logic for missing methods.

What do you think?

[rhenium]

Implementing #private? on the base class requires ruby/openssl to keep track of which components an OpenSSL::PKey::PKey contains, because OpenSSL doesn't provide a generic public API to check whether an EVP_PKEY is a private key, a public key, or key parameters only (openssl/openssl#9467).

I looked into this before in #527, but eventually gave up because supporting a wide range of OpenSSL versions seemed to make it too fragile and complicated. I think we could revisit this after dropping support for OpenSSL 1.1.1 (IOW, after removing OpenSSL::PKey::{RSA,DSA,DH}#set_* methods).

{RSA,DSA,DH,EC}#to_pem uses several different non-standard formats (which OpenSSL's man pages refer to as traditional). PKey#private_to_pem always uses PKCS#8 regardless of the pkey type. My recommendation is to use #private_to_pem whenever possible, though you might need to continue using #to_pem on RSA/DSA/DH/EC for backwards compatibility.

Or I want OpenSSL::PKey::ML-DSA or OpenSSL::PKey::ML-DSA-NN classes to have the missing methods.

FWIW the reason we haven't added dedicated classes for ML-DSA, ML-KEM, Ed25519, X25519, etc. is simply that we haven't needed to introduce methods specific to these pkey types.

[junaruga]

Thanks for the explaining the context.

For #private?, I experimented the following method for ML-DSA-44 with Ruby compiled with OpenSSL 4.1 dev.

diff --git a/lib/openssl/pkey.rb b/lib/openssl/pkey.rb
index 39871e1..9a9d43a 100644
--- a/lib/openssl/pkey.rb
+++ b/lib/openssl/pkey.rb
@@ -7,6 +7,15 @@
 require_relative 'marshal'

 module OpenSSL::PKey
+  class PKey
+    def private?
+      private_to_pem
+      true
+    rescue OpenSSL::PKey::PKeyError
+      false
+    end
+  end
+
   # Alias of PKeyError. Before version 4.0.0, this was a subclass of PKeyError.
   DHError = PKeyError

test.rb

require 'openssl'

private_key = OpenSSL::PKey.generate_key("ML-DSA-44")
p private_key.private?
public_key = OpenSSL::PKey.read(private_key.public_to_der)
p public_key.private?

$ which ruby
~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/bin/ruby

$ ruby -I lib/ test.rb
true
false

The test.rb worked as I expected in OpenSSL 4.1 dev. This behavior depends on the PEM_write_bio_PKCS8PrivateKey returning 0 when the argument pkey is public key in my case. But this situation is fragile when considering other OpenSSL versions and other SSL library LibreSSL and AWS-LC, right?

openssl/ext/openssl/ossl_pkey.c

Lines 966 to 970 in 27c59a4

	if (!PEM_write_bio_PKCS8PrivateKey(bio, pkey, enc, NULL, 0,
	ossl_pem_passwd_cb, (void *)pass)) {
	BIO_free(bio);
	ossl_raise(ePKeyError, "PEM_write_bio_PKCS8PrivateKey");
	}

For the #to_pem, Yes, the reason I keep using #to_pem is because of the backward compatibility.
But I think I need to refactor the current existing #to_pem to replace it with #public_to_pem or #private_to_pem in ruby/rubygems before working on ruby/rubygems#9697.

[rhenium]

I don't think we can implement a generic #private? this way in ruby/openssl. Although ruby/openssl currently has little support for third-party providers, a pkey backed by an HSM-based provider wouldn't allow exporting the private key with #private_to_pem.