bundle install affected by DST Root CA X3 expiration

[LizhangX]

Hi,

We are getting the below issue when doing bundle install with jruby today since the root CA is expired.
https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190

There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. For information about OpenSSL certificates, see http://bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and change 'https' to 'http'.
Added /opt/jruby/lib/ruby/stdlib/rubygems/ssl_certs/rubygems.org/AddTrustExternalCARoot.pem to X509 store
Added /opt/jruby/lib/ruby/stdlib/rubygems/ssl_certs/rubygems.global.ssl.fastly.net/DigiCertHighAssuranceEVRootCA.pem to X509 store
Added /opt/jruby/lib/ruby/stdlib/rubygems/ssl_certs/index.rubygems.org/GlobalSignRootCA.pem to X509 store
SSL verification error at depth 2: unable to get local issuer certificate (20)
You must add /O=Digital Signature Trust Co./CN=DST Root CA X3 to your local trusted store

We added the debug printing about ssl verification result above.

So basically it doesn't trust the first cert it finds in the chain even tho it is in the keystore.
DST Root CA X3 is not there, just like in the OS store.
If we add it to the key store, we are getting the below error.

SSL verification error at depth 3: certificate has expired (10)
Certificate /O=Digital Signature Trust Co./CN=DST Root CA X3 expired at 2021-09-30T14:01:15Z

Maybe this flag X509_V_FLAG_TRUSTED_FIRST should be added here https://github.com/jruby/jruby-openssl/blob/master/src/main/java/org/jruby/ext/openssl/x509store/VerifyParameter.java#L153-L162

We tried bundle install with pure ruby and it is working fine. Then we noticed that jRuby is using this module instead of openssl.
Could we have some suggestions here please?

[Sharpie]

Could be the same root cause as #232 given both involve iterating through potential certificate chains in order to find the valid one.

[igcherkaev]

Could be the same root cause as #232 given both involve iterating through potential certificate chains in order to find the valid one.

Hmm, I don't think so... In the case of expired DST Root CA X3 cert, it doesn't stop validation even though ISRG Root X1 cert is in the keystore and trusted. The X509_V_FLAG_TRUSTED_FIRST flag is probably not being set when jopenssl asks OpenSSL to verify the chain of certs, so once it sees that ISRG Root X1 is issued by DST Root CA X3, it wants to validate it as well.

curl, for example, made that change a while ago: curl/curl@e2de2d5

But what's bothering me is that OpenSSL 1.1.0 (which is the case with jruby docker images based on debian 9 stretch) says it enables that flag by default. So in theory this issue should not have popped up today at all, but it is what it is. Unless jopenssl overrides openssl's defaults somehow.

Like, here, maybe: https://github.com/jruby/jruby-openssl/blob/master/src/main/java/org/jruby/ext/openssl/x509store/VerifyParameter.java#L76

or, here: https://github.com/jruby/jruby-openssl/blob/master/src/main/java/org/jruby/ext/openssl/x509store/VerifyParameter.java#L286

Because if you look here: https://github.com/openssl/openssl/blob/278260bfa238aefef5a1abe2043d2f812c3a4bd5/crypto/x509/x509_vpm.c#L513 - openssl sets that flag there, and it's probably not zero.

[bjoernhaeuser]

If Android 2-7 compatibility is not an issue for your certificates, you can also tell certbot to only use the new certificate chain:

--preferred-chain "ISRG Root X1"

(you maybe need to also set --force-renewal to get a net certificate. More information are here.

[rhanka]

The problem appears also if you just use this lib to get when you don't manage certificates (cf above logstash issues), so it is blocking.

[jsvd]

Here is a reproducer for this issue: https://gist.github.com/jsvd/cdd0b7421bac7b2a00d64b644eba12c4

[jsvd]

Update: removing the expired root certificate from the JDK's truststore doesn't workaround the issue.
I used the following code to test it: https://gist.githubusercontent.com/jsvd/c72a51f5ffda234a0d12a590fb03ce82/raw/8fbddab6689880a49b8481c1121355713db85339/gistfile1.txt

The steps where:

1. ran the script and observed failed verify with error string: certificate has expired
2. deleted expired root cert from JDK's truststore: bin/keytool -cacerts -delete -alias 'o=digital signature trust co.,cn=dst root ca x3' (error code 10)
3. ran the script again and observed failed verify with different error string: unable to get local issuer certificate (error code: 20)

[igcherkaev]

Update: removing the expired root certificate from the JDK's truststore doesn't workaround the issue. I used the following code to test it: https://gist.githubusercontent.com/jsvd/c72a51f5ffda234a0d12a590fb03ce82/raw/8fbddab6689880a49b8481c1121355713db85339/gistfile1.txt

The steps where:

1. ran the script and observed failed verify with error string: certificate has expired
2. deleted expired root cert from JDK's truststore: bin/keytool -cacerts -delete -alias 'o=digital signature trust co.,cn=dst root ca x3' (error code 10)
3. ran the script again and observed failed verify with different error string: unable to get local issuer certificate (error code: 20)

Yeah, we tried the same, got same results. Hence, we created this issue as there seems to be no workaround available.

[tedt10e]

Not sure this is related but if it is something about DST Root CA X3 expiration for JRuby app with Nginx server, this may help jruby/jruby#6872 (comment)

[jordan-thoms]

Yeah, we're still being blocked by this - any progress on fixing this issue?