test_pkey_ec.rb test failures in OpenSSL FIPS

[junaruga]

I am trying to fix the test failures in test/openssl/test_pkey_ec.rb now in OpenSSL FIPS on the ruby/openssl latest master branch f4b8dacc75d61142b7b4e0142898b2fecbb131b9, and openssl/openssl latest master branch cf712830b7b5a20a768a1fc5f78dc48841b7617f.

Test failures

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  bundle exec rake debug
...
ruby 3.3.0dev (2023-05-30T12:39:26Z master 30b960ba34) [x86_64-linux]
OpenSSL::OPENSSL_VERSION: OpenSSL 3.2.0-dev 
OpenSSL::OPENSSL_LIBRARY_VERSION: OpenSSL 3.2.0-dev 
OpenSSL::OPENSSL_VERSION_NUMBER: 30200000
OpenSSL::LIBRESSL_VERSION_NUMBER: undefined
FIPS enabled: true

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl test/openssl/test_pkey_ec.rb
Loaded suite test/openssl/test_pkey_ec
Started
E
===================================================================================================================================================================================================================
Error: test_ECPrivateKey_encrypted(OpenSSL::TestEC): OpenSSL::PKey::ECError: invalid curve name
test/openssl/test_pkey_ec.rb:247:in `initialize'
test/openssl/test_pkey_ec.rb:247:in `new'
test/openssl/test_pkey_ec.rb:247:in `test_ECPrivateKey_encrypted'
     244:     0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=
     245:     -----END EC PRIVATE KEY-----
     246:     EOF
  => 247:     key = OpenSSL::PKey::EC.new(pem, "abcdef")
     248:     assert_same_ec p256, key
     249:     key = OpenSSL::PKey::EC.new(pem) { "abcdef" }
     250:     assert_same_ec p256, key
===================================================================================================================================================================================================================
E
===================================================================================================================================================================================================================
Error: test_ec_key(OpenSSL::TestEC): NoMethodError: undefined method `filter_backtrace' for module Test
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:188:in `block in assert_nothing_raised'
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:26:in `block in message'
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:190:in `rescue in assert_nothing_raised'
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:181:in `assert_nothing_raised'
test/openssl/test_pkey_ec.rb:19:in `block in test_ec_key'
     16:       key = OpenSSL::PKey::EC.generate(curve_name)
     17:       assert_predicate key, :private?
     18:       assert_predicate key, :public?
  => 19:       assert_nothing_raised { key.check_key }
     20:     end
     21: 
     22:     key1 = OpenSSL::PKey::EC.generate("prime256v1")
test/openssl/test_pkey_ec.rb:11:in `each'
test/openssl/test_pkey_ec.rb:11:in `test_ec_key'
===================================================================================================================================================================================================================
Finished in 0.044746681 seconds.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
16 tests, 124 assertions, 0 failures, 2 errors, 0 pendings, 0 omissions, 0 notifications
87.5% passed
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
357.57 tests/s, 2771.16 assertions/s

A minimal reproducer

For the test/openssl/test_pkey_ec.rb:19, below is a minimal reproducer.

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key'
-e:1:in `check_key': EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError)
	from -e:1:in `<main>'

Debug with GDB

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  gdb --args ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key'
...
(gdb) set environment LD_LIBRARY_PATH /home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/lib
(gdb) b ossl_ec_key_check_key
(gdb) r
...
(gdb) n
551	        if (EVP_PKEY_check(pctx) != 1) {
(gdb) p EVP_PKEY_check(pctx)
$1 = 0
(gdb) f
#0  ossl_ec_key_check_key (self=140737044094120)
    at ../../../../ext/openssl/ossl_pkey_ec.c:551
551	        if (EVP_PKEY_check(pctx) != 1) {
(gdb) n
552	            EVP_PKEY_CTX_free(pctx)
(gdb) p pctx
$2 = (EVP_PKEY_CTX *) 0x7e17d0
(gdb) p *pctx
$3 = {operation = 0, libctx = 0x0, propquery = 0x0, 
  keytype = 0x7fffe54c830a "id-ecPublicKey", keymgmt = 0x7c5410, op = {keymgmt = {
      genctx = 0x0}, kex = {exchange = 0x0, algctx = 0x0}, sig = {signature = 0x0, 
      algctx = 0x0}, ciph = {cipher = 0x0, algctx = 0x0}, encap = {kem = 0x0, 
      algctx = 0x0}}, cached_parameters = {dist_id_name = 0x0, dist_id = 0x0, 
    dist_id_len = 0, dist_id_set = 0}, app_data = 0x0, pkey_gencb = 0x0, 
  keygen_info = 0x0, keygen_info_count = 0, legacy_keytype = 408, pmeth = 0x0, 
  engine = 0x0, pkey = 0x7e0570, peerkey = 0x0, data = 0x0, flag_call_digest_custom = 0, 
  rsa_pubexp = 0x0}
(gdb) n
553	            ossl_raise(eECError, "EVP_PKEY_check");

The EVP_PKEY_check(pctx) returns 0 in the line below. And it seems that causes the EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError). Do you know why this happens?

openssl/ext/openssl/ossl_pkey_ec.c

Lines 551 to 554 in f4b8dac

	if (EVP_PKEY_check(pctx) != 1) {
	EVP_PKEY_CTX_free(pctx);
	ossl_raise(eECError, "EVP_PKEY_check");
	}

[junaruga]

Error: test_ec_key(OpenSSL::TestEC): NoMethodError: undefined method `filter_backtrace' for module Test
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:188:in `block in assert_nothing_raised'

For the error above, where does the Test.filter_backtrace come from?

openssl/test/lib/core_assertions.rb

Lines 185 to 190 in f4b8dac

	rescue *(args.empty? ? Exception : args) => e
	msg = message(msg) {
	"Exception raised:\n<#{mu_pp(e)}>\n""Backtrace:\n" <<
	Test.filter_backtrace(e.backtrace).map{|frame| " #{frame}"}.join("\n")
	}
	raise Test::Unit::AssertionFailedError, msg.call, e.backtrace

[junaruga]

For the error above, where does the Test.filter_backtrace come from?

The test/lib/core_assertions.rb including the Test.filter_backtrace was added by the 520601e by @hsbt. @hsbt, do you know why the error by the Test.filter_backtrace happens?

And I am seeing the same name method filter_backtrace in the test-unit. Is it just co-incidence?
https://github.com/test-unit/test-unit/blob/8cd57617a3cb470985cf5768691e10ca72928a1c/lib/test/unit/util/backtracefilter.rb#L20-L55

[rhenium]

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key'
-e:1:in `check_key': EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError)
	from -e:1:in `<main>'

secp112r1 isn't allowed in FIPS 140.

[rhenium]

Error: test_ec_key(OpenSSL::TestEC): NoMethodError: undefined method `filter_backtrace' for module Test
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:188:in `block in assert_nothing_raised'

For the error above, where does the Test.filter_backtrace come from?

openssl/test/lib/core_assertions.rb

Lines 185 to 190 in f4b8dac

	rescue *(args.empty? ? Exception : args) => e
	msg = message(msg) {
	"Exception raised:\n<#{mu_pp(e)}>\n""Backtrace:\n" <<
	Test.filter_backtrace(e.backtrace).map{|frame| " #{frame}"}.join("\n")
	}
	raise Test::Unit::AssertionFailedError, msg.call, e.backtrace

First, we should switch to using https://github.com/ruby/test-unit-ruby-core instead of embedding core_assertions.rb. This is just not yet worked on.

The same issue is in ruby/test-unit-ruby-core. Test.filter_backtrace appears to be defined in ruby/ruby's tool/lib/test/unit.rb. It could be included in, too, but there doesn't seem a need for CoreAssertions to define assert_nothing_raised since test-unit already provides this assertion method.

[hsbt]

ruby/test-unit-ruby-core#2 (comment) is helpful for openssl?

[rhenium]

Yes, test-unit-ruby-core 1.0.2 fixed assert_nothing_raised for me. PR #673 lets this repository use the gem instead of embedding core_assertions.rb.

[junaruga]

@hsbt @rhenium thanks! On the latest ruby/opessl including the #673, the assert_nothing_raised fails properly.

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  bundle exec ruby -I./lib -ropenssl test/openssl/test_pkey_ec.rb
Loaded suite test/openssl/test_pkey_ec
Started
E
==========================================================================================
Error: test_ECPrivateKey_encrypted(OpenSSL::TestEC): OpenSSL::PKey::ECError: invalid curve name
test/openssl/test_pkey_ec.rb:247:in `initialize'
test/openssl/test_pkey_ec.rb:247:in `new'
test/openssl/test_pkey_ec.rb:247:in `test_ECPrivateKey_encrypted'
     244:     0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=
     245:     -----END EC PRIVATE KEY-----
     246:     EOF
  => 247:     key = OpenSSL::PKey::EC.new(pem, "abcdef")
     248:     assert_same_ec p256, key
     249:     key = OpenSSL::PKey::EC.new(pem) { "abcdef" }
     250:     assert_same_ec p256, key
==========================================================================================
F
==========================================================================================
Failure: test_ec_key(OpenSSL::TestEC):
  Exception raised:
  <#<OpenSSL::PKey::ECError: EVP_PKEY_check: initialization error>>
  Backtrace:
    test/openssl/test_pkey_ec.rb:19:in `check_key'
    test/openssl/test_pkey_ec.rb:19:in `block (2 levels) in test_ec_key'.
test/openssl/test_pkey_ec.rb:19:in `check_key'
test/openssl/test_pkey_ec.rb:19:in `block (2 levels) in test_ec_key'
     16:       key = OpenSSL::PKey::EC.generate(curve_name)
     17:       assert_predicate key, :private?
     18:       assert_predicate key, :public?
  => 19:       assert_nothing_raised { key.check_key }
     20:     end
     21: 
     22:     key1 = OpenSSL::PKey::EC.generate("prime256v1")
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.3.0+0/gems/test-unit-ruby-core-1.0.2/lib/core_assertions.rb:219:in `assert_nothing_raised'
test/openssl/test_pkey_ec.rb:19:in `block in test_ec_key'
test/openssl/test_pkey_ec.rb:11:in `each'
test/openssl/test_pkey_ec.rb:11:in `test_ec_key'
==========================================================================================
Finished in 0.050731255 seconds.
------------------------------------------------------------------------------------------
16 tests, 124 assertions, 1 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
87.5% passed
------------------------------------------------------------------------------------------
315.39 tests/s, 2444.25 assertions/s

[junaruga]

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key'
-e:1:in `check_key': EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError)
	from -e:1:in `<main>'

secp112r1 isn't allowed in FIPS 140.

How did you know that? Could you share a document link that you checked for that?

[junaruga]

secp112r1 isn't allowed in FIPS 140.

How did you know that? Could you share a document link that you checked for that?

Is it https://www.openssl.org/source/ - The OpenSSL 3.0.0 or 3.0.8 security policy document?

[rhenium]

secp112r1, defined on a 112-bit finite field, would provide at most 56 bits of security. I don't know how to cite the specifications for FIPS 140, but it must definitely be prohibited for any use.

The test case in question is about OpenSSL::PKey::EC.builtin_curves. I don't think the current assertions make much sense anyway. I think it can use something like:

diff --git a/test/openssl/test_pkey_ec.rb b/test/openssl/test_pkey_ec.rb
index e5fef940a6c3..ab777a8b48a4 100644
--- a/test/openssl/test_pkey_ec.rb
+++ b/test/openssl/test_pkey_ec.rb
@@ -5,20 +5,6 @@
 
 class OpenSSL::TestEC < OpenSSL::PKeyTestCase
   def test_ec_key
-    builtin_curves = OpenSSL::PKey::EC.builtin_curves
-    assert_not_empty builtin_curves
-
-    builtin_curves.each do |curve_name, comment|
-      # Oakley curves and X25519 are not suitable for signing and causes
-      # FIPS-selftest failure on some environment, so skip for now.
-      next if ["Oakley", "X25519"].any? { |n| curve_name.start_with?(n) }
-
-      key = OpenSSL::PKey::EC.generate(curve_name)
-      assert_predicate key, :private?
-      assert_predicate key, :public?
-      assert_nothing_raised { key.check_key }
-    end
-
     key1 = OpenSSL::PKey::EC.generate("prime256v1")
 
     # PKey is immutable in OpenSSL >= 3.0; constructing an empty EC object is
@@ -49,6 +35,17 @@ def test_ec_key
     end
   end
 
+  def test_builtin_curves
+    builtin_curves = OpenSSL::PKey::EC.builtin_curves
+    assert_not_empty builtin_curves
+    assert_equal 2, builtin_curves[0].size
+    assert_kind_of String, builtin_curves[0][0]
+    assert_kind_of String, builtin_curves[0][1]
+
+    builtin_curve_names = builtin_curves.map { |name, comment| name }
+    assert_include builtin_curve_names, "prime256v1"
+  end
+
   def test_generate
     assert_raise(OpenSSL::PKey::ECError) { OpenSSL::PKey::EC.generate("non-existent") }
     g = OpenSSL::PKey::EC::Group.new("prime256v1")

[rhenium]

I submitted the above patch as #675. This should work without and with the FIPS provider without reducing test coverage.

The other failure is probably the same issue as #643, the usage of MD5.

[junaruga]

secp112r1, defined on a 112-bit finite field, would provide at most 56 bits of security. I don't know how to cite the specifications for FIPS 140, but it must definitely be prohibited for any use.

The test case in question is about OpenSSL::PKey::EC.builtin_curves. I don't think the current assertions make much sense anyway. I think it can use something like:

diff --git a/test/openssl/test_pkey_ec.rb b/test/openssl/test_pkey_ec.rb
index e5fef940a6c3..ab777a8b48a4 100644
--- a/test/openssl/test_pkey_ec.rb
+++ b/test/openssl/test_pkey_ec.rb
@@ -5,20 +5,6 @@
 
 class OpenSSL::TestEC < OpenSSL::PKeyTestCase
   def test_ec_key
-    builtin_curves = OpenSSL::PKey::EC.builtin_curves
-    assert_not_empty builtin_curves
-
-    builtin_curves.each do |curve_name, comment|
-      # Oakley curves and X25519 are not suitable for signing and causes
-      # FIPS-selftest failure on some environment, so skip for now.
-      next if ["Oakley", "X25519"].any? { |n| curve_name.start_with?(n) }
-
-      key = OpenSSL::PKey::EC.generate(curve_name)
-      assert_predicate key, :private?
-      assert_predicate key, :public?
-      assert_nothing_raised { key.check_key }
-    end
-
     key1 = OpenSSL::PKey::EC.generate("prime256v1")
 
     # PKey is immutable in OpenSSL >= 3.0; constructing an empty EC object is
@@ -49,6 +35,17 @@ def test_ec_key
     end
   end
 
+  def test_builtin_curves
+    builtin_curves = OpenSSL::PKey::EC.builtin_curves
+    assert_not_empty builtin_curves
+    assert_equal 2, builtin_curves[0].size
+    assert_kind_of String, builtin_curves[0][0]
+    assert_kind_of String, builtin_curves[0][1]
+
+    builtin_curve_names = builtin_curves.map { |name, comment| name }
+    assert_include builtin_curve_names, "prime256v1"
+  end
+
   def test_generate
     assert_raise(OpenSSL::PKey::ECError) { OpenSSL::PKey::EC.generate("non-existent") }
     g = OpenSSL::PKey::EC::Group.new("prime256v1")

I see. The solution looks okay to me. I may notice that one issue related to the OpenSSL::PKey::EC.builtin_curves (the used EC_get_builtin_curves). The issue is it seems to me that the result of the EC_get_builtin_curves(curves, crv_len) in the code below looks same in both FIPS and non-FIPS cases.

https://github.com/ruby/openssl/blob/7c34a439bae71f110dd40be35454b68bd2b82e37/ext/openssl/ossl_pkey_ec.c#L885C9-L885C9

FIPS

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves.size'
82

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves[0]'
["secp112r1", "SECG/WTLS curve over a 112 bit prime field"]

Non-FIPS

$ ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves.size'
82

$ ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves[0]'
["secp112r1", "SECG/WTLS curve over a 112 bit prime field"]

This is weird to me. Because the EC_get_builtin_curves returns the curve_list.

https://github.com/openssl/openssl/blob/a2608e4bc430d6216bbf36f50a29278e8759103a/crypto/ec/ec_curve.c#L3316-L3331

And it seems that the curve_list is declared conditionally in both FIPS (FIPS_MODULE is defined) at crypto/ec/ec_curve.c#L2827 and non-FIPS (FIPS_MODULE not defined) crypto/ec/ec_curve.c#L2901. So, I expect the result of the EC_get_builtin_curves is different between FIPS and non-FIPS cases. I might see that FIPS_MODULE macro is not defined in the FIPS case. I plan to ask this to OpenSSL project.