Update return rules for null-safety?

[eernstg]

Cf. dart-lang/sdk#41324.

We may wish to revisit the rules about return e; because the rules in the language specification mention Null specifically in one case, and may need to treat Never specifically in other cases).

For instance, Future<Never> f() async => null as dynamic; is a compile-time error with the current rules, with null-safety. But this is because Future<dynamic> is no longer assignable to Future<Never> (so the wrapping with Future gives the new assignability a distinction that the old assignability did not have). In contrast, Never f1() => null as dynamic; is OK (with null-safety as well as without) because dynamic is assignable to Never.

In both cases we could argue that an expression of type dynamic can be trusted to throw or loop (that is, to not complete, with or without a value) just as well as it could be trusted to return an object of an arbitrary type which is required by the context, or to some type that has a specific member with a specific signature. And we do trust the dynamic expression for all those purposes.

So it seems consistent to allow return e; when e has type dynamic, both in a synchronous non-generator function and in an asynchronous non-generator function.

If we do this then we'd need to adjust the language specification to avoid the wrapping in Future<...>. This change should be a no-op semantically for now (same meaning), but it should ensure that the rules are still specifying the intended behavior when we introduce null-safety into the language specification.

@munificent, @lrhn, @leafpetersen, WDYT?

[leafpetersen]

I'm not sure I see any reason to change any of these. I do think we should clean up the errors/warnings around return statements as part of this release, since they're in an odd state. We specified them as warnings to avoid breakage, but then some or all of them got implemented as errors in the analyzer, so currently the analyzer and CFE differ. We should take this opportunity to make all of them consistent.

[eernstg]

The point is that null-safety introduces an anomaly:

// In a legacy library.
int f1() => 42 as dynamic; // OK.
Future<int> f2() async => 42 as dynamic; // OK.

// In a library where null-safety is enabled.
int f3() => 42 as dynamic; // OK.
Future<int> f4() async => 42 as dynamic; // Error.

The intuition is that we always "return" 42 as dynamic, and there is no difference between the situation where we check (statically) that it is OK to cast from dynamic to int, both in the case where we are actually returning it (f1) and in the case where we are using the "returned" value to complete the Future<int> which was returned previously (f2).

However, we have specified the async case in terms of a wrapped type: We require that in the case where flatten(dynamic) is not void (which is true) that Future<flatten(dynamic)> (that is, Future<dynamic>) must be assignable to the return type Future<int>. The latter is not true with null-safety, so we get an error in f4.

But note that we are never performing an assignment from an expression of type Future<dynamic> to a variable of type Future<int>!

So we're performing a cast from dynamic to int in the process where the "returned" value completes the future, which is OK also with null-safety. But this is accidentally specified in terms of a cast from Future<dynamic> to Future<int>, which is not allowed with null-safety.

We might just as well have used the following (arguably more natural) specification:

It is a compile-time error if $s$ is \code{\RETURN{} $e$;},
\flatten{S} is not \VOID,
and \flatten{S} is not assignable to the element type of $f$
(\ref{functions}).

The only reason why we did not do this was that we did not have a definition of the relevant concept when \section{Return} was last updated.

We have the notion of the 'element type of a generator function', and we'd need a corresponding concept of something like the 'element type of an asynchronous non-generator function'. With return type Future<T> or FutureOr<T> that's T, and with return type void or dynamic it is dynamic.

So I'd recommend that we define this new concept, and make that change in \section{Return}. We then avoid making f4 an error, maintain the conceptual consistency (f3 as well as f4 is actually about a cast from dynamic to int, so they should both be an error, or none of them should be an error). We then also avoid breaking code just because we used an approach in the specification which is needlessly convoluted, which produces this error by accident, and which does not reflect the actual semantics.

I closed #913 (a PR with two changes), because the first change was subsumed by existing rules about return statements in the language specification, and the second change was intended to specify that certain errors (here) are indeed errors — but at this point I'd prefer that we change the specification such that they are not errors after all.

[lrhn]

I agree in principle that a return in an async function should work like any other FutureOr<flatten(returnType)> context.

The context for f4 is FutureOr<int>, and that's what we should be downcasting dynamic to (then check whether it's a future and await it if so).

[leafpetersen]

[Edit: Never mind - user error. The below does fail at runtime.]

The context for f4 is FutureOr<int>, and that's what we should be downcasting dynamic to (then check whether it's a future and await it if so).

I don't think you really mean that do you? If so then this program (which currently works) becomes an error:

Future<int> foo() async => (Future<num>.value(3) as dynamic);

void main() async {
  print(await foo());
}

[leafpetersen]

As I said at the high level - I'm not averse to cleaning up the errors and warnings around returns. The spec was added after the fact to try to unify the de facto rules implemented by the two front ends into something consistent and minimally breaking - hence all of the odd exceptions around Null.

If someone wants to take this on (@eernstg?) then what needs to happen is:

• A proposal for the new rules
• Matching updates to the appropriate tests (see tests/language_2/invalid_returns.*), plus possibly new tests covering any new behavior not otherwise covered.
• Once we are agreed, implementation issues for the front ends.

[lrhn]

It's never simple with async functions!

Assume we have an async function with declared return type R, and let T be flatten(R). (We can call this the "element type" of the async function, or something).

First of all: The simplest solution to specify, is to disallow returning Future<T>. Require the user to always write the await if they want to return the result of a future, not return the future directly. That would reduce the problem to ... zero. Context type of the return statement expression is the element type of the function. That is all. If you want to await, you use an explicit await with the same rules as all other awaits. That's both prettier and simpler and quite a lot harder to migrate.
(Not impossible, though, and I hope we'll do it some day).

Assuming we want to keep returning futures, the typing is tricky.

Let's have a return statement of the form return e; where e has static type S.
The run-time behavior is:

• Evaluate e to a value o.
• If o implements Future<T>, await o and let v be the resulting value.
• Otherwise let v be o.
• If o does not implement T, throw a TypeError (this handles implicit downcast from dynamic).
• Complete the returned future with v.

That can be written as something like:

return let tmp = e in tmp is Future<T> ? await tmp : (tmp implicit-downcast-to T);`.

Statically typing that to disallow cases where tmp as T would definitely throw is not easy.
I guess we can simply ignore the tmp is Future<T> case because that one will definitely generate something of type T.
So, we need to detect the cases where the tmp is not implicitly downcastable to T, and that branch is still potentially reachable.

1. We could say that S must be assignable to FutureOr<T>.

That will disallow return Future<dynamic>.value(2); in an async function returning Future<int>, even though return await Future<dynamic>.value(2); is allowed.
Do we want to allow this case?
As currently specified, we will not await that future because it does not implement Future<int>. We don't await all futures, only those where awaiting them will give a valid return value.
Maybe allowing it is not actually desirable.

The static return expression types allowed by this rule, for the element type T, are:

• X, X <: T
• Future<X>, X <: T
• FutureOr<T>, X <: T
• dynamic – Assignable to anything.

That's the same as saying: return e; is valid iff S is assignable to T or flatten(S) is a subtype of T.

2. Or we could say that a return e; where e has type S is allowed iff S is assignable to T or flatten(S) is assignable to T.

That's almost the same as S being assignable to FutureOr<T>, but allows the case above because flatten(Future<dynamic>) is dynamic which is assignable to int.
Which is not necessarily an improvement, since it doesn't match run-time behavior.

The static return expression types allowed by the rule, for the element type T, are:

• X, X <: T
• Future<X>, X <: T
• FutureOr<T>, X <: T
• dynamic – Assignable to anything.
• Future<dynamic>
• FutureOr<dynamic>

These are the types which are either subtypes of T or are dynamic, or which flattens to something which is a subtype of T or is dynamic.

So, let's look at some cases:

import "dart:async";

var f1 = Future<int>.value(0);
var f2 = Future<Future<int>>.value(f1);
var f3 = Future<Future<Future<int>>>.value(f2);
var fo1 = Future<dynamic>.value(2);

Future<int> foo(int x) async {
  if (x == 0) return x;
  if (x == 1) return x as dynamic;
  if (x == 2) return x as FutureOr<int>;
  if (x == 3) return x as FutureOr<dynamic>;
  if (x == 10) return f1;
  if (x == 11) return f1 as dynamic;
  if (x == 12) return f1 as Future<dynamic>; // Disallowed by 1.
  if (x == 13) return f1 as FutureOr<int>; 
  if (x == 14) return f1 as FutureOr<dynamic>; // Disallowed by 1.
  if (x == 15) return f1 as FutureOr<FutureOr<int>>; // Disallowed by 1 and 2.
  if (x == 16) return f1 as FutureOr<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (x == 20) return fo1;  // Disallowed by 1. Throws at runtime.
  if (x == 21) return fo1 as dynamic;
  if (x == 22) return fo1 as FutureOr<dynamic>;  // Disallowed by 1.
  return throw "not";
}

Future<Future<int>> bar(int y) async {
  if (y == 0) return f1; // Valid. Disallowed by analyzer.
  if (y == 1) return f1 as Future<dynamic>; // Disallowed by 1.
  if (y == 2) return f1 as FutureOr<int>; // Disallowed by 1 and 2. Allowed by CFE.
  if (y == 3) return f1 as FutureOr<dynamic>; // Disallowed by 1.
  if (y == 10) return f2;
  if (y == 11) return f2 as dynamic;
  if (y == 12) return f2 as Future<dynamic>; // Disallowed by 1.
  if (y == 13) return f2 as FutureOr<dynamic>; // Disallowed by 1.
  if (y == 14) return f2 as Future<Future<dynamic>>;  // Disallowed by 1 and 2.
  if (y == 15) return f2 as FutureOr<Future<dynamic>>; // Disallowed by 1 and 2
  if (y == 16) return f2 as Future<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (y == 17) return f2 as FutureOr<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  return throw "not";
}

Future<FutureOr<int>> baz(int z) async {
  if (z == 0) return z;
  if (z == 1) return z as dynamic;
  if (z == 2) return z as FutureOr<int>;
  if (z == 3) return z as FutureOr<FutureOr<int>>;
  if (z == 10) return f1;
  if (z == 11) return f1 as dynamic;
  if (z == 12) return f1 as Future<dynamic>;  // Disallowed by 1.
  if (z == 13) return f1 as FutureOr<int>;
  if (z == 14) return f1 as FutureOr<dynamic>; // Disallowed by 1.
  if (z == 15) return f1 as Future<FutureOr<int>>;
  if (z == 16) return f1 as FutureOr<Future<int>>;
  if (z == 17) return f1 as FutureOr<FutureOr<int>>;
  if (z == 18) return f1 as Future<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (z == 19) return f1 as FutureOr<Future<dynamic>>; // Disallowed by 1, 2 and Analyzer.
  if (z == 20) return f1 as FutureOr<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (z == 30) return f2; // Disallowed by CFE
  if (z == 31) return f2 as dynamic;
  if (z == 32) return f2 as Future<dynamic>; // Disallowed by 1.
  if (z == 33) return f2 as Future<Future<int>>; // Disallowed by CFE
  if (z == 34) return f2 as Future<Future<dynamic>>; // Disallowed by 1 and Analyzer
  if (z == 35) return f2 as FutureOr<dynamic>; // Disallowed by 1.
  if (z == 36) return f2 as FutureOr<Future<int>>;
  if (z == 37) return f2 as Future<FutureOr<int>>;
  if (z == 38) return f2 as FutureOr<FutureOr<int>>;
  if (z == 39) return f2 as FutureOr<Future<dynamic>>; // Disallowed by 1, 2, and Analyzer
  if (z == 40) return f2 as Future<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (z == 41) return f2 as FutureOr<FutureOr<dynamic>>; // Disallowed by 1 and 2.
  if (z == 50) return f3; // Disallowed by Analyzer
  return throw "not";
}

Currently the cases without specific comments are accepted by both CFE and analyzer (checked in DartPad). The disallowed by 1/2 are referring to the rules above in a setting with no implicit downcasts except from dynamic (assignable-to means either subtype-of or dynamic)

[eernstg]

Nice analysis, @lrhn!

It matches my perspective on the situation in PR #930, where I've chosen the approach that you call model 2, and it could obviously be changed to model 1 by editing a couple of words.

I introduced the notion of "the future value type" of an async function, which is the same thing as the "element type" that you mention. However, with null-safety it is not exactly the same thing as flatten(R) where R is the declared return type (in particular Future<int>? f() async {...} is allowed unless we introduce some extra errors for certain return types, and that function should have a future value type of int, not int?).

Model 2 does allow for an extra case compared to model 1: we can return an expression of type Future<dynamic> in an async function with future value type T which is not dynamic, and it will not be an error, we await it and check dynamically whether the completion has the required type T. This seems conceptually consistent to me: It is a separate feature that return may await a future, so awaiting those futures that implement Future<U> such that U is assignable to the future value type is the obvious criterion, and that's model 2. We will need to adjust the tools such that they allow this statically, and such that they generate code to await a Future<dynamic>.

We can also return an expression e of type dynamic, so how do we treat the object obtained from evaluation of e if it has type Future<dynamic> or Future<Object?>? In this case there is no static expectation that we might obtain an instance of any particular type, so any reference to assignability (being a static concept) is questionable. It seems clearly wrong to await the Future<Object?> (we do not even have assignability), and it seems inconsistent to treat the Future<Object?> differently from the Future<dynamic> at run time.

So I'd argue that with model 2 we should only await a Future<dynamic> based on the static type, because the run-time treatment of an object o whose type contains dynamic does not otherwise differ from the run-time treatment where those occurrences of dynamic in the run-time type of o have been replaced by another top type.

So the dynamic semantics should continue to check for a Future<T> (where T is the future value type of the function) and await that, and otherwise not await any futures. If it turns out to be a Future<dynamic> completed with a T we still fail at run time (because we did not await that T), and the motivation is that the special powers of dynamic only apply during static analysis.

Conversely, model 1 introduces the conceptual oddity that the object obtained directly by evaluation of the returned expression is treated differently from an object which is awaited: For the former we allow static type dynamic and check the actual type at run-time, for the latter we insist that the static type is safe.

The whole point of this issue is that this is new, and it is a breaking change. Also, the above oddity indicates to me that this breaking change is poorly justified conceptually.

So that's the reason why I chose model 2 when I wrote PR #930. Note that PR #930 fixes an issue with the existing spec text, and is otherwise a no-op (no changes), as long as we consider pre-null-safety Dart; but it prepares the spec for null-safety because it avoids introducing the above mentioned breaking change "by accident" when null-safety is introduced.