Possible GC bugs triggered on arm64

[mkustermann]

From the flakiness dashboard we have failures

like this from e688753:

/============================================================================\
| vm/dart/generic_field_invocation_test broke (Pass -> Crash, expected Pass) |
\============================================================================/

--- Command "vm" (took 09.000378s):
DART_CONFIGURATION=ReleaseXARM64 out/ReleaseXARM64/dart --no-lazy-dispatchers --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/runtime/tests/vm/dart/generic_field_invocation_test.dart

exit code:
-6

stderr:
Class Id: 0
../../runtime/vm/raw_object.cc: 340: error: unreachable code
version=2.6.0-edge.e6887536aadc7fbd1990448989601cee0224958d (Thu Sep 12 01:20:05 2019 +0000) on "linux_arm64"
thread=2331, isolate=kernel-service(0xaaaad48f2400)
  pc 0x0000aaaaaff0b60c fp 0x0000ffff954fe560 dart::Profiler::DumpStackTrace(void*)
-- End of DumpStackTrace

--- Re-run this test:
python tools/test.py -n dartk-linux-release-arm64 vm/dart/generic_field_invocation_test

There seems to be a core file available for the build here:

[mkustermann]

/cc @rmacnak-google @mraleph @a-siva

[mkustermann]

@rmacnak-google Would you be interested in looking at this?

[rmacnak-google]

I've left this running in a loop in gdb on my ARM64 board, but it's looking unlikely that it will reproduce.

[mraleph]

@mkustermann @rmacnak-google have you tried sanity checking the stack trace in the core to see if this can be a data race? Those ARM64 bots are notorious for triggering those.

[rmacnak-google]

Somehow I had overlooked the core file.

The main thread finished a compilation whose allocation triggered the beginning of concurrent marking. Two markers have started. One is iterating through new space and somehow finds itself in the middle of a ForwardingCorpse (a filler object to make TLABs iterable). The other marker finished its half of the root set and has started to drain the mark stack. This is during a safepoint, so I think it's unlikely that this is a data race; the two marker threads should see all previous writes to the heap because they are synchronized through the safepoint monitors, but it's possible a data race could happen just between the markers somehow.

[rmacnak-google]

standalone_2/no_lazy_dispatchers_test is also flaky failing in a similar way. These tests have in common that they set --no-lazy-dispatchers. With --no-lazy-dispatcher set, the code generated by PushArrayOfArguments runs much more often, and I see there is a barrier present in the other architectures and absent in ARM64, so that might the cause.

[rmacnak-google]

Both generic_field_invocation_test and no_lazy_dispatchers_test have no recent crashes on the flakiness dashboard.