GoogleTestSecurityManager replacement for tests that call System.exit

[cushon]

Java tests run by Bazel have historically installed the GoogleTestSecurityManager, which prevents the tests from calling System.exit (and Runtime.exit and Runtime.halt). The idea is to prevent a test that calls System.exit(0) from spuriously passing, or failing in a hard-to-debug way if it reports another exit code.

The SecurityManager has been deprecated for several JDK releases, and is being removed in JDK 24: https://openjdk.org/jeps/486

To avoid breaking Java tests running on JDK 24, the GoogleTestSecurityManager is now disabled by default: 366c4ba

This bug tracks providing replacement functionality for tests that call System.exit. There are a number of possibilities:

• There's some related discussion in Bazel@HEAD on JDK 17: GoogleTestSecurityManager WARNING: System::setSecurityManager will be removed in a future release #14502, in particular in Bazel@HEAD on JDK 17: GoogleTestSecurityManager WARNING: System::setSecurityManager will be removed in a future release #14502 (comment) @fmeum suggested using "the combination of TEST_PREMATURE_EXIT_FILE and the improved JDK logging for System.exit calls."
• Runtime#addShutdownHook could be used to intercept calls to System.exit, print where they happened, and ensure the test shuts down with a non-zero exit code.
• The JEP mentions this use-case and suggests using a java.lang.instrument agent to do bytecode rewriting on tests and block calls to System.exit. That would work, but it does require some maintenance and runtime overhead to install an agent and do the bytecode rewriting for every test execution.
• There's an OpenJDK FR for Ability to intercept or disable System::exit, which is not planned for JDK 24.

There's been some internal discussion about this that leaned towards the addShutdownHook approach.

[fmeum]

Runtime#addShutdownHook could be used to intercept calls to System.exit, print where they happened, and ensure the test shuts down with a non-zero exit code

Does this work based on the assumption that shutdown hooks are executed on a call stack that also contains the System.exit call? It looks like the API docs don't guarantee that, but it does seem to be the case.

[cushon]

Runtime#addShutdownHook could be used to intercept calls to System.exit, print where they happened, and ensure the test shuts down with a non-zero exit code

Does this work based on the assumption that shutdown hooks are executed on a call stack that also contains the System.exit call? It looks like the API docs don't guarantee that, but it does seem to be the case.

One idea was to look at thread stacks before executing, to ensure it worked if the hook was executed on a different thread.