OpenSSL 3: OpenSSL.fips_mode returns false in FIPS enabled environment

[junaruga]

Here is another FIPS specific issue with OpenSSL 3 on RHEL 9.1 with FIPS mode enabled. The reason is because the macro OPENSSL_FIPS is not used in OpenSSL 3 any more.

The OpenSSL.fips_mode returns "false" in the environment.

# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.1 (Plow)

# fips-mode-setup --check
FIPS mode is enabled.

# echo $?
0

# openssl version
OpenSSL 3.0.1 14 Dec 2021 (Library: OpenSSL 3.0.1 14 Dec 2021)

# ruby -v
ruby 3.0.4p208 (2022-04-12 revision 3fa771dded) [x86_64-linux]

# gem list | grep openssl
openssl (default: 2.2.1)

# ruby -e 'require "openssl"; p OpenSSL.fips_mode'
false

Here is the part of the code.

openssl/ext/openssl/ossl.c

Lines 417 to 428 in fbb24fd

	static VALUE
	ossl_fips_mode_get(VALUE self)
	{
	#ifdef OPENSSL_FIPS
	VALUE enabled;
	enabled = FIPS_mode() ? Qtrue : Qfalse;
	return enabled;
	#else
	return Qfalse;
	#endif
	}

Here is the gdb log.

# gdb -q -ex "set breakpoint pending on" --ex "set debuginfod enabled on" --args ruby -e 'require "openssl"; p OpenSSL.fips_mode'

(gdb) b main
Breakpoint 1 at 0x1130: file ./main.c, line 38.

(gdb) b ossl_fips_mode_get
Function "ossl_fips_mode_get" not defined.
Breakpoint 2 (ossl_fips_mode_get) pending.

(gdb) r

<..snip..>

(gdb) f
#0  ossl_fips_mode_get (self=140737288759560)
    at /usr/src/debug/ruby-3.0.4-160.el9_0.x86_64/ext/openssl/ossl.c:406
406	{

(gdb) l
401	 * call-seq:
402	 *   OpenSSL.fips_mode -> true | false
403	 */
404	static VALUE
405	ossl_fips_mode_get(VALUE self)
406	{
407	
408	#ifdef OPENSSL_FIPS
409	    VALUE enabled;
410	    enabled = FIPS_mode() ? Qtrue : Qfalse;

(gdb) n
413	    return Qfalse;

(gdb) bt
#0  ossl_fips_mode_get (self=140737288759560)
    at /usr/src/debug/ruby-3.0.4-160.el9_0.x86_64/ext/openssl/ossl.c:413
#1  0x00007ffff7e64302 in vm_call_cfunc_with_frame (ec=0x55555555de90, 
    reg_cfp=0x7ffff77fdfa0, calling=<optimized out>)
    at /usr/src/debug/ruby-3.0.4-160.el9_0.x86_64/vm_insnhelper.c:2931
#2  0x00007ffff7e68017 in vm_sendish (ec=0x55555555de90, reg_cfp=0x7ffff77fdfa0, 
    cd=0x55555575d410, block_handler=<optimized out>, 
    method_explorer=<optimized out>)
    at /usr/src/debug/ruby-3.0.4-160.el9_0.x86_64/vm_callinfo.h:336
#3  0x00007ffff7e6c4e8 in vm_exec_core (ec=0x55555555de90, initial=0)
    at /usr/src/debug/ruby-3.0.4-160.el9_0.x86_64/insns.def:789
#4  0x00007ffff7e83f30 in rb_vm_exec (ec=0x55555555de90, 
    mjit_enable_p=<optimized out>)
    at /usr/src/debug/ruby-3.0.4-160.el9_0.x86_64/vm.c:2179
#5  0x00007ffff7ce833f in rb_ec_exec_node (ec=ec@entry=0x55555555de90, 
    n=n@entry=0x7ffff419c9a8)
    at /usr/src/debug/ruby-3.0.4-160.el9_0.x86_64/eval.c:317
#6  0x00007ffff7ce845a in ruby_run_node (n=0x7ffff419c9a8)
    at /usr/src/debug/ruby-3.0.4-160.el9_0.x86_64/eval.c:375
#7  0x000055555555518f in main (argc=<optimized out>, argv=<optimized out>)
    at ./main.c:50

(gdb) c
Continuing.
false
[Inferior 1 (process 13802) exited normally]

It seems that OPENSSL_FIPS implementation was deleted at the commit openssl/openssl@b53338c.

diff --git a/Configure b/Configure
index 4404963aa7..f6d5a7cfd3 100755
--- a/Configure
+++ b/Configure
...
-    if ($config{fips}) {
-       push @{$config{openssl_other_defines}}, "OPENSSL_FIPS";
-    }
-
...

I checked the FIPS related documents on OpenSSL repository. We need to add an implementation to make the Ruby OpenSSL work with OpenSSL 3 with the FIPS enabled environment.

$ pwd
/home/jaruga/git/openssl

$ find doc/ -name "*fips*"
doc/man7/fips_module.pod
doc/man5/fips_config.pod
doc/man1/openssl-fipsinstall.pod.in

$ find doc/ -name "*FIPS*"
doc/man7/OSSL_PROVIDER-FIPS.pod

The man 7 fips_module: https://github.com/openssl/openssl/blob/master/doc/man7/fips_module.pod

Note that the old functions FIPS_mode() and FIPS_mode_set() are no longer present so you must remove them from your application if you use them.
Applications written to use the OpenSSL 3.0 FIPS module should not use any legacy APIs or features that avoid the FIPS module. Specifically this includes:

• Low level cryptographic APIs (use the high level APIs, such as EVP, instead)
• Engines
• Any functions that create or modify custom "METHODS" (for example EVP_MD_meth_new(), EVP_CIPHER_meth_new(), EVP_PKEY_meth_new(), RSA_meth_new(), EC_KEY_METHOD_new(), etc.)

All of the above APIs are deprecated in OpenSSL 3.0 - so a simple rule is to avoid using all deprecated functions. See migration_guide(7) for a list of deprecated functions.

The man 7 migration_guide: https://github.com/openssl/openssl/blob/master/doc/man7/migration_guide.pod

[junaruga]

This issue was fixed by #608 (and #621 fixing an issue by #608 in OpenSSL 1.1).

I tested this issue with the latest ruby/openssl master branch 82da58e319edf653241b855ea7ac1b59af9d1118 and with OpenSSL 3.0.8 compiled from the source with FIPS mode option in Fedora Linux 37 below.

Install OpenSSL from the source.

$ git clone https://github.com/openssl/openssl.git

$ git checkout openssl-3.0.8

$ ./Configure \
  --prefix=$HOME/.local/openssl-3.0.8-fips \
  --libdir=lib \
  shared \
  enable-fips

$ make -j$(nproc)

$ make install

ruby/openssl

$ bundle exec rake compile -- --with-openssl-dir=$HOME/.local/openssl-3.0.8-fips

Check the linked libraries.

$ ldd lib/openssl.so 
	linux-vdso.so.1 (0x00007fff211db000)
	libruby.so.3.2 => /usr/local/ruby-3.2.1/lib/libruby.so.3.2 (0x00007f6402c00000)
	libssl.so.3 => /home/jaruga/.local/openssl-3.0.8-fips/lib/libssl.so.3 (0x00007f6403317000)
	libcrypto.so.3 => /home/jaruga/.local/openssl-3.0.8-fips/lib/libcrypto.so.3 (0x00007f6402600000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f6402b20000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f6402423000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f64032e3000)
	libgmp.so.10 => /lib64/libgmp.so.10 (0x00007f6402a7b000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f64032a9000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6403289000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f640341f000)

Here is the OpenSSL config file to run on the FIPS mode.

$ cat $HOME/.local/openssl-3.0.8-fips/ssl/openssl_fips.cnf
config_diagnostics = 1
openssl_conf = openssl_init

.include /home/jaruga/.local/openssl-3.0.8-fips/ssl/fipsmodule.cnf
#.include ./fipsmodule.cnf

[openssl_init]
providers = provider_sect
alg_section = algorithm_sect

[provider_sect]
fips = fips_sect
base = base_sect

[base_sect]
activate = 1

[algorithm_sect]
default_properties = fips=yes

The command below returns the true.

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.0.8-fips/ssl/openssl_fips.cnf \
  ruby -I./lib -e 'require "openssl"; p OpenSSL.fips_mode'
true

Without specifying the OpenSSL config file, the command below returns the false as expected, as it doesn't load the "fips" provider and "default_properties = fips=yes".

$ ruby -I./lib -e 'require "openssl"; p OpenSSL.fips_mode'
false

[akostadinov]

FIPS is enabled in the kernel. It doesn't make sense to run kernel in FIPS mode but OpenSSL - not. Also it doesn't make sense to run openssl in FIPS mode but kernel not. This is because the system will not be FIPS compliant in either of these configurations.

So I think that it makes more sense to check the kernel configuration and enable/disable FIPS in openssl based on this.

$ cat /proc/sys/crypto/fips_enabled
1

[junaruga]

@akostadinov How did you know that FIPS is only enabled in the kernel? In my understanding, the /proc/sys/crypto/fips_enabled is a Fedora/RHEL downstream OpenSSL specific feature. I was told from the maintainer that this patch manages it. The policy of the Fedora/RHEL for the FIPS mode is different from the upstream.

Here is the upstream document about the FIPS mode.
https://github.com/openssl/openssl/blob/master/README-FIPS.md

[akostadinov]

@junaruga , that was my guess but now I just checked Ubuntu FAQ [1] and it also does refers to /proc/sys/crypto/fips_enabled. So it seems to me that it is not something limited to Fedora/RHEL.

I didn't know that the OpenSSL check for /proc/sys/crypto/fips_enabled is a separate RHEL patch. To me mixing FIPS and non-FIPS enabled components doesn't make sense. I wonder if OpenSSL makes this choice automatically on other operating systems or they decided to rely solely on configuration everywhere.

In either case I understand why the openssl gem may stay with upstream defaults. Thank you for the links.

[1] https://ubuntu.com/security/certifications/docs/fips-faq

[junaruga]

Thank you for the link explaining the /proc/sys/crypto/fips_enabled in Ubuntu. I assume that there was a discussion if the OpenSSL had an OS level switch to enable/disable FIPS mode in the OpenSSL in the OpenSSL project. In my understanding, packagers in the Fedora project usually avoid the downstream patches as much as possible. And I have seen that Linux or *BSD distro packagers of a package (component) sometimes work together in the upstream. I am not surprised that the /proc/sys/crypto/fips_enabled is used in both Federa/RHEL and Ubuntu.

And yes, as you said, I guess that mixing FIPS and non-FIPS components doesn't happen in the real use case. In the case of Fedora and RHEL, according to the document in RHEL 9, the FIPS mode is enabled in the OS level by the commands below.

# fips-mode-setup --enable

# reboot

However, only enabling the OpenSSL FIPS mode in the non-FIPS mode OS is useful for a testing purpose like this case. In the Fedora specific patch above, the environment variable OPENSSL_FORCE_FIPS_MODE=1 that works on Fedora 38 or later is a convenient switch to enable OpenSSL FIPS mode in the non-FIPS mode OS. And as far as I know, the OPENSSL_FORCE_FIPS_MODE=0 doesn't set OpenSSL non-FIPS mode in the FIPS mode OS. So, it doesn't make the OS unsafe.

As a note, you can change the value of the /proc/sys/crypto/fips_enabled by the root user without rebooting OS. Here is the script that I created in the past. I only tested it on RHEL 9.