Failures on Update to latest package:json_rpc_2

[zichangg]

There are new test failures on Update to latest package:json_rpc_2.

The tests

service/external_service_notification_invocation_test/dds RuntimeError (expected Pass)

are failing on configurations

dartkp-win-release-x64

log: https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8872333091123864224/+/steps/test_results/0/logs/new_test_failures__logs_/0

[natebosch]

This appears flaky, the same test is passing in other configurations.

The error does look plausibly related to my changes in that package, but without being able to reproduce this locally I'm not quite sure.

Should I roll this back in the mean time?

[natebosch]

I'm not sure how to proceed with this. I spent some time trying to see if I could reproduce a similar failure outside of the SDK and I was able to track down at least one way to cause the behavior I saw in the logs. That was fixed with dart-archive/json_rpc_2#65

When trying to sync that it surfaced a different bug in dds that was being masked by bugs in json_rpc_2. I fixed that bug in https://dart-review.googlesource.com/c/sdk/+/158220

But now I'm stuck. When I try to sync today I can get a green run on this bot, but it's not clear to me if that is sufficient.

Here is a green run: https://ci.chromium.org/p/dart/builders/try/vm-kernel-precomp-win-release-x64-try/1872?

There are some things here I don't understand at all.

• The "Test Results" link shows:

  dartkp-win-release-x64
  Pass -> RuntimeError (expected Pass)
  service/external_service_disappear_test/dds
  

• The "Test Results" link does not show a message about service/external_service_notification_invocation_test/dds going from RuntimeError to Pass
• The results.json link shows both service/external_service_disappear_test/dds and service/external_service_notification_invocation_test/dds as "Pass".

One thing that is interesting, flaky.json does contain service/external_service_notification_invocation_test/dds.

I have been completely unable to see this failure locally.

So I have some questions:

@whesse

• Why is there a discrepancy between results.json and https://dart-ci.firebaseapp.com/cl/158206/3 ? Is it showing me the failure still from a previous run?
• How should I interpret this try result? Is it safe to submit my CL? Would submitting that be considered enough to close this issue?
• How can I see the output from the run(s) that failed?
• Is there any way for me to more narrowly check a patch to see if it fixes a given test?

@bkonyi - can you help me dig here? I think you might own some of these tests?

• Do you know if this test has been flaky in the past?
• Do you know why this failure might only show up on windows? Are you able to run this test locally and see an error?

[bkonyi]

@bkonyi - can you help me dig here? I think you might own some of these tests?

• Do you know if this test has been flaky in the past?
• Do you know why this failure might only show up on windows? Are you able to run this test locally and see an error?

Service tests are notoriously flaky on Windows. This is an older test that's being run against DDS, so it's possible that the extra latency introduced by communicating with the service via DDS has made the test more flaky. It's also hard to tell what the actual failure is caused by since we check the error code before we check the message so the error message is never written to the console.

I think you're safe to land, but if you go ahead and swap lines 84 and 85 in runtime/observatory/tests/service/external_service_disappear_test.dart we'll get a more useful error output if we hit it again.

[natebosch]

@bkonyi - The test I'm most interested in currently is service/external_service_notification_invocation_test/dds

This is the failure that I am intending to address with dart-archive/json_rpc_2#65.

The stack trace was:

  Bad state: The client is closed.

  package:json_rpc_2/src/client.dart 152                      Client._send
  package:json_rpc_2/src/client.dart 140                      Client.sendNotification
  package:json_rpc_2/src/peer.dart 102                        Peer.sendNotification
  package:dds/src/client.dart 74                              _DartDevelopmentServiceClient.sendNotification

DDS is trying to do the right thing here and check whether the peer is closed, but a bug in json_rpc_2 meant that a Peer could appear open, but it's Client was closed and so the call fails.

sdk/pkg/dds/lib/src/client.dart

Lines 71 to 74 in 3b6aeb4

	if (_clientPeer.isClosed) {
	return;
	}
	_clientPeer.sendNotification(method, parameters);

I was able to track down one way that can lead to this situation and I fixed that. (Side note, my repro required a StreamChannel that does not fit the contract around close guarantees which suggests that either DDS or maybe just that test has a misbehaving StreamChannel). Since I can't see the logs and I can't reproduce a failure locally I don't know if they flakiness in this test is still the same thing (and I haven't fixed it), or something unrelated...

The other test service/external_service_disappear_test/dds was fixed ahead of time by https://dart-review.googlesource.com/c/sdk/+/158220 and I'm hoping @whesse will confirm that there is something going wrong in the UI that it still showing it as a new failure.