lib/openssl/hmac.rb:55:in `initialize': EVP_PKEY_new_mac_key: malloc failure (OpenSSL::HMACError) with -Werror flag

[junaruga]

On the current master branch 22e601a, I got an error. I found this error on the Ubuntu 22.04 Ruby head, 3.2 and 3.1 cases on GitHub Actions. But the error didn't happen on Ubuntu 22.04 Ruby 3.0 and 2.7 cases. The CI result on my fork repository is here, and 7 pending tests too. The 7 pending test didn't happen in Ubuntu 22.04 Ruby 3.0 case.

$ gcc --version
gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4)
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ which ruby
~/.local/ruby-30b960ba34-debug/bin/ruby

$ ruby -v
ruby 3.3.0dev (2023-05-30T12:39:26Z master 30b960ba34) [x86_64-linux]

$ LD_LIBRARY_PATH=$HOME/.local/openssl-3.0.8-debug/lib/ \
  $HOME/.local/openssl-3.0.8-debug/bin/openssl version 
OpenSSL 3.0.8 7 Feb 2023 (Library: OpenSSL 3.0.8 7 Feb 2023)

$ MAKEFLAGS="V=1" \
  bundle exec rake compile -- \
  --with-openssl-dir=$HOME/.local/openssl-3.0.8-debug \
  --with-cflags="-Wall -Wextra -Wdeprecated-declarations -Wdiv-by-zero -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wold-style-definition -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef -Werror"

$ ruby -I lib -ropenssl -e 'OpenSSL::HMAC.hexdigest("SHA256", "", "test")'
/home/jaruga/var/git/ruby/openssl/lib/openssl/hmac.rb:55:in `initialize': EVP_PKEY_new_mac_key: malloc failure (OpenSSL::HMACError)
	from /home/jaruga/var/git/ruby/openssl/lib/openssl/hmac.rb:55:in `new'
	from /home/jaruga/var/git/ruby/openssl/lib/openssl/hmac.rb:55:in `hexdigest'
	from -e:1:in `<main>'

This error didn't happen without the -Werror. So, I assume that one of the warning flags cause this error.

$ rm -rf tmp/ lib/openssl.so

$ MAKEFLAGS="V=1" \
  bundle exec rake compile -- \
  --with-openssl-dir=$HOME/.local/openssl-3.0.8-debug \
  --with-cflags="-Wall -Wextra -Wdeprecated-declarations -Wdiv-by-zero -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wold-style-definition -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef"

$ ruby -I lib -ropenssl -e 'OpenSSL::HMAC.hexdigest("SHA256", "", "test")'

$ echo $?
0

Debug

I debugged with debug flags with GDB.

$ rm -rf tmp/ lib/openssl.so

MAKEFLAGS="V=1" \
  bundle exec rake compile -- \
  --with-openssl-dir=$HOME/.local/openssl-3.0.8-debug \
  --with-cflags="-Wall -Wextra -Wdeprecated-declarations -Wdiv-by-zero -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wold-style-definition -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef -Werror -O0 -g3 -ggdb3 -gdwarf-5"

$ ruby -I lib -ropenssl -e 'OpenSSL::HMAC.hexdigest("SHA256", "", "test")'
/home/jaruga/var/git/ruby/openssl/lib/openssl/hmac.rb:55:in `initialize': EVP_PKEY_new_mac_key: malloc failure (OpenSSL::HMACError)
	from /home/jaruga/var/git/ruby/openssl/lib/openssl/hmac.rb:55:in `new'
	from /home/jaruga/var/git/ruby/openssl/lib/openssl/hmac.rb:55:in `hexdigest'
	from -e:1:in `<main>'

The EVP_PKEY_new_mac_key returns NULL
in the ossl_hmac_initialize in ext/openssl/ossl_hmac.c:107.

$ gdb --args ruby -I lib -ropenssl -e 'OpenSSL::HMAC.hexdigest("SHA256", "", "test")'
...<snip>...
(gdb) l
105	        ossl_raise(eHMACError, "EVP_PKEY_new_raw_private_key");
106	#else
107	    pkey = EVP_PKEY_new_mac_key(EVP_PKEY_HMAC, NULL,
108	                                (unsigned char *)RSTRING_PTR(key),
109	                                RSTRING_LENINT(key));
110	    if (!pkey)
111	        ossl_raise(eHMACError, "EVP_PKEY_new_mac_key");
112	#endif
113	    if (EVP_DigestSignInit(ctx, NULL, ossl_evp_get_digestbyname(digest),
114	                           NULL, pkey) != 1) {
(gdb) f
#0  ossl_hmac_initialize (self=140737045325080, key=140737045325360, digest=140737045325440) at ../../../../ext/openssl/ossl_hmac.c:110
110	    if (!pkey)
(gdb) p pkey
$4 = (EVP_PKEY *) 0x0
(gdb) p (unsigned char *)RSTRING_PTR(key)
$5 = (unsigned char *) 0x7fffe597e648 ""
(gdb) p RSTRING_LENINT(key)
$6 = 0

[junaruga]

I will check if this issue happens with Ruby 3.0.

[rhenium]

The EVP_PKEY_new_mac_key returns NULL
in the ossl_hmac_initialize in ext/openssl/ossl_hmac.c:107.

This is a known issue in OpenSSL 3.0 when the HMAC key is empty: openssl/openssl#13089

However, this code is supposed to be unreachable with OpenSSL >= 3.0 (#538).

[junaruga]

CC: @cwjenkins

Thank you for providing the info!

I checked the CI results, and the "ubuntu-22.04 (ruby) 3.1" case fails and "ubuntu-22.04 (ruby) 3.0" passes. So, I assumed that this is a Ruby version specific issue. Because it seemed that it was executed in the same environment.
https://github.com/junaruga/openssl/actions/runs/5125132987/jobs/9217902633

But it was a wrong assumption. I checked the used OpenSSL versions now. And I found the difference of used OpenSSL versions.

• "ubuntu-22.04 (ruby) 3.1": OpenSSL 3.0.2 is used. The issue happened.
  https://github.com/junaruga/openssl/actions/runs/5125132987/jobs/9217902633#step:8:19
• "ubuntu-22.04 (ruby) 3.0": OpenSSL 1.1.1t is used. The issue didn't happen.
  https://github.com/junaruga/openssl/actions/runs/5125132987/jobs/9217902535#step:8:19

[rhenium]

I reproduced it with RUBY_OPENSSL_EXTCFLAGS=-Werror rake compile. mkmf.log says:

have_func: checking for EVP_PKEY_new_raw_private_key(0, NULL, (unsigned char *)"", 0) in openssl/evp.h... -------------------- no

LD_LIBRARY_PATH=.:/opt/ruby/master/lib "gcc -o conftest -I/opt/ruby/master/include/ruby-3.3.0+0/x86_64-linux -I/opt/ruby/master/include/ruby-3.3.0+0/ruby/backward -I/opt/ruby/master/include/ruby-3.3.0+0 -I../../../../ext/openssl -I/opt/openssl/3.0-stable/include    -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wdiv-by-zero -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wold-style-definition -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef  -fPIC -Werror conftest.c  -L. -L/opt/ruby/master/lib -Wl,-rpath,/opt/ruby/master/lib -L/opt/openssl/3.0-stable/lib -Wl,-rpath,/opt/openssl/3.0-stable/lib -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed    -lssl -lcrypto  -Wl,-rpath,/opt/ruby/master/lib -L/opt/ruby/master/lib -lruby -lssl -lcrypto  -lm -lpthread  -lc"
conftest.c: In function ‘t’:
conftest.c:17:30: error: ‘s1’ may be used uninitialized [-Werror=maybe-uninitialized]
   17 | int t(void) { char s1[1024]; EVP_PKEY_new_raw_private_key(0, NULL, (unsigned char *)s1, 0); return 0; }
      |                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from conftest.c:3:
/opt/openssl/3.0-stable/include/openssl/evp.h:1810:11: note: by argument 3 of type ‘const unsigned char *’ to ‘EVP_PKEY_new_raw_private_key’ declared here
 1810 | EVP_PKEY *EVP_PKEY_new_raw_private_key(int type, ENGINE *e,
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
conftest.c:17:20: note: ‘s1’ declared here
   17 | int t(void) { char s1[1024]; EVP_PKEY_new_raw_private_key(0, NULL, (unsigned char *)s1, 0); return 0; }
      |                    ^~
At top level:
cc1: note: unrecognized command-line option ‘-Wno-self-assign’ may have been intended to silence earlier diagnostics
cc1: note: unrecognized command-line option ‘-Wno-parentheses-equality’ may have been intended to silence earlier diagnostics
cc1: note: unrecognized command-line option ‘-Wno-constant-logical-operand’ may have been intended to silence earlier diagnostics
cc1: all warnings being treated as errors
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <openssl/evp.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     int (* volatile tp)(void)=(int (*)(void))&t;
11:     printf("%d", (*tp)());
12:   }
13: 
14:   return !!argv[argc];
15: }
16: 
17: int t(void) { char s1[1024]; EVP_PKEY_new_raw_private_key(0, NULL, (unsigned char *)s1, 0); return 0; }
/* end */

The checking program used by have_func generally is not expected to be a perfectly valid program, so I think we should not apply -Werror to it. I think we can move the append_cflags logic in extconf.rb to after all checking code, right before create_makefile.

[junaruga]

The checking program used by have_func generally is not expected to be a perfectly valid program, so I think we should not apply -Werror to it. I think we can move the append_cflags logic in extconf.rb to after all checking code, right before create_makefile.

I see. The error happened in the have_func. And I agree with your idea moving to the right before create_makefile. Do you like to send the pull-request by yourself?

[rhenium]

Already! Haha #633

[junaruga]

All right. Thanks. :)

[rhenium]

I checked the CI results, and the "ubuntu-22.04 (ruby) 3.1" case fails and "ubuntu-22.04 (ruby) 3.0" passes.

This is probably because the checking program was not producing a warning on Ruby 3.0.

A change in Ruby 3.1 (ruby/ruby@cb5a41c) replaced the "" in the argument with a char s1[1024]. I don't really understand the intention, however.

However, I think it will likely happen again that a new warning starts to appear in the checking program (e.g., by OpenSSL converting a function to a macro), I think the safest workaround is to not apply -Werror.

[junaruga]

This is probably because the checking program was not producing a warning on Ruby 3.0.
A change in Ruby 3.1 (ruby/ruby@cb5a41c) replaced the "" in the argument with a char s1[1024]. I don't really understand the intention, however.

Sorry, I don't understand what the change in Ruby 3.1 (ruby/ruby@cb5a41c) is related to about the CI results I provided above.

In the "ubuntu-22.04 (ruby) 3.1" case, the rake compile with -Werror passed without the compiler warnings, and the rake test got the 1 error and 7 pendings that always happen with the -Werror in the case. In my understanding, the cause is by use of the OpenSSL 3.0.

And in the "ubuntu-22.04 (ruby) 3.1" case, the rake compile with -Werror passed without the compiler warnings and the rake test also passed.

[rhenium]

Sorry for the misleading quotation. I missed the fact the "ubuntu-22.04 (ruby) 3.0" uses OpenSSL 1.1.1.

The "openssl-3.0.8" configuration uses Ruby 3.0 + OpenSSL 3.0 + -Werror and it passed without failures or errors.

This is an issue when all of these are met: Ruby >= 3.1, OpenSSL >= 3.0, and -Werror.

[junaruga]

All right. That makes sense. 👍🏼

[junaruga]

I don't see this issue on the CI environments on the #631. O assume that the #633 fixed it. Anyway, I think we can close this ticket.