dart` segfaults on `dart test --disable-dart-dev`

[matanlurey]

Foreword: After talking to @jakemac53 and @bkonyi, it seems like these commands were not intended to work together.

Filing a bug here for the CLI team to do their thing :)

---

Original issue: flutter/flutter#154263

As of flutter/engine#54811, I am moving dart tooling tests written with a homegrown test runner to use dart test (and package:test). As part of that move, instead of enumerating test/*_test.dart files manually, and invoking them with dart {%s}, we're invoking dart test (with the current working directory temporarily changed to the package path).

Concretely:

dart_tests = glob.glob('%s/test/*_test.dart' % package_path)
if not dart_tests:
  raise Exception('No tests found for Dart package at %s' % package_path)
for dart_test_file in dart_tests:
  opts = ['--disable-dart-dev', dart_test_file] + extra_opts
  yield EngineExecutableTask(
      build_dir, os.path.join('dart-sdk', 'bin', 'dart'), None, flags=opts, cwd=package_path
  )

Becomes:

opts = ['test', '--disable-dart-dev']
yield EngineExecutableTask(
    build_dir, os.path.join('dart-sdk', 'bin', 'dart'), None, flags=opts, cwd=package_path
)

However, this command causes a segfault on both MacOS and Linux platforms.

To reproduce this easily, check out (and build) the latest version of the engine, and do the following:

# Assuming host_debug_unopt_arm64 is built, but substitute another pre-built if needed.
cd tools/header_guard_check
../../../out/host_debug_unopt_arm64/dart-sdk/bin/dart test --disable-dart-dev

===== CRASH =====
si_signo=Segmentation fault: 11(11), si_code=SEGV_ACCERR(2), si_addr=0x0
version=3.6.0-180.0.dev (dev) (Mon Aug 26 05:02:58 2024 -0700) on "macos_arm64"
pid=8421, thread=259, isolate_group=(nil)(0x0), isolate=(nil)(0x0)
os=macos, arch=arm64, comp=no, sim=no
isolate_instructions=0, vm_instructions=1029d9360
fp=16d456d10, sp=16d456cf0, pc=19c683904
  pc 0x000000019c683904 fp 0x000000016d456d10 _platform_strlen+0x4
  pc 0x0000000102e669ac fp 0x000000016d456d40 /Users/matanl/Developer/engine/src/out/host_debug_unopt_arm64/dart-sdk/bin/dart+0x4be9ac
  pc 0x00000001029d627c fp 0x000000016d456de0 /Users/matanl/Developer/engine/src/out/host_debug_unopt_arm64/dart-sdk/bin/dart+0x2e27c
  pc 0x00000001029d6100 fp 0x000000016d456e40 /Users/matanl/Developer/engine/src/out/host_debug_unopt_arm64/dart-sdk/bin/dart+0x2e100
  pc 0x00000001029d74cc fp 0x000000016d456f20 /Users/matanl/Developer/engine/src/out/host_debug_unopt_arm64/dart-sdk/bin/dart+0x2f4cc
  pc 0x00000001029d70f4 fp 0x000000016d456fd0 /Users/matanl/Developer/engine/src/out/host_debug_unopt_arm64/dart-sdk/bin/dart+0x2f0f4
  pc 0x00000001029d7f08 fp 0x000000016d457140 /Users/matanl/Developer/engine/src/out/host_debug_unopt_arm64/dart-sdk/bin/dart+0x2ff08
  pc 0x00000001029d7008 fp 0x000000016d457160 /Users/matanl/Developer/engine/src/out/host_debug_unopt_arm64/dart-sdk/bin/dart+0x2f008
  pc 0x000000019c2cb154 fp 0x000000016d457390 start+0x9ac
-- End of DumpStackTrace
zsh: abort      ../../../out/host_debug_unopt_arm64/dart-sdk/bin/dart test --disable-dart-dev

The SDK-shipped Dart SDK also crashes:

cd tools/header_guard_check
../../prebuilts/macos-arm64/dart-sdk/bin/dart test --disable-dart-dev

I believe we use --disable-dart-dev as its not something that adds value to CI runs, and in the past has caused minor forms of instability and/or log spam, but I can't be certain. For now I can workaround this by not using --disable-dart-dev, but I figured I should file this for posterity.