Simple dependent types

[JukkaL]

[This is based on a discussion between me and @gvanrossum last week. I'm writing this down so that we won't forget about this.]

Several stdlib functions have signatures where the return type depends on the value of an argument. Examples:

• open returns an IO[str] or IO[bytes] depending on whether the mode argument has 'b' as a substring.
• subprocess.check_output returns a str or bytes depending on the value of the boolean universal_newlines argument.

A simple way to support these would be to introduce a few additional types that work with str and bool literals:

• A string pattern type that can describe the contents of a string literal, perhaps using a regular expression.
• Types for False and True.

For example, if we had FalseType and TrueType, we could write a function whose return value depends on the value of a boolean argument:

@overload
def f(x: FalseType) -> bytes: ...
@overload
def f(x: TrueType) -> str: ...

def f(x: Any) -> Any:
    if x:
        return 'x'
    else:
        return b'x'

reveal_type(f(False))  # bytes
reveal_type(f(True)  # str

Not sure about this one:

...
x = bool('')
reveal_type(f(x))  # Union[str, bytes] since we don't know the boolean value statically?

We could also have a generic class where a generic type argument depends on the value of an argument to __init__. Not sure how we should represent these -- here is one potential approach:

class C(Generic[AnyStr]):
    @overload
    @returntype('C[str]')
    def __init__(self, x: TrueType) -> None: ...
    @overload
    @returntype('C[bytes]')
    def __init__(self, x: FalseType) -> None: ...
    ...

reveal_type(C(True))  # C[str]

Alternatively, we could use __new__, but this would be problematic if there is no corresponding method defined at runtime.

[elazarg]

I think we can have LiteralType as in LiteralType['w'], LiteralType[True] etc. I think it will be better than special-casing True/False/string. Maybe we will get simple constant propagation "falling out" of type aliases (which might be helpful for e.g. NamedTuple)

I think regex will be problematic. How do you determine whether they overlap? String literals seems to suffice.

(And I don't think it counts as dependent type since it has little to do with runtime).

[gvanrossum]

The mode string syntax for open() is more complicated than just 'r', 'w', etc. Hence the regex.

I find LiteralType[True] pretty verbose; also it's more general than we want to implement (we really don't want this to be a big project, just enough to handle the cases @JukkaL mentioned).

If this doesn't count as a dependent type maybe you can explain your definition of dependent types? And what would you call this?

[elazarg]

I'm not sure going into definitions is helpful, but the difference is (I guess) the reason you call it "Simple" dependent types - there nothing like def foo(x: int, y: bar[x]) -> str if x == 5 else bool. There's only information immediately available at compile time. Instead of defining True to be a literal of type bool, we define it as being a literal of type TrueType which is a subtype of (or coercible to) bool. I don't see the difference between this and the kind of types we already have. Just because the type has precisely single value you have precise information about its runtime value, but this is already the case with None, Void and Tuple[()].

However, I might be wrong. I am also too easily dragged into issues of terminology.

[sixolet]

Possibilities for when you can't statically determine the overload variant because your argument isn't literal enough: * Union the alternatives you've got * Use the return type of the implementation (if any) (simpler, I think) * Use `Any` (simplest, I think)

…

On Mon, Mar 27, 2017 at 2:37 PM, Guido van Rossum ***@***.***> wrote: The mode string syntax for open() is more complicated than just 'r', 'w', etc. Hence the regex. I find LiteralType[True] pretty verbose; also it's more general than we want to implement (we really don't want this to be a big project, just enough to handle the cases @JukkaL <https://github.com/JukkaL> mentioned). If this doesn't count as a dependent type maybe you can explain your definition of dependent types? And what would you call this? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3062 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABjs4Q_pmhJYnbeMNorZzX3OAIZpKeCzks5rqCw1gaJpZM4Mqfj6> .

[elazarg]

I suspect special cases will continue to appear and build, and one day we will have to do this big project - which I have the feeling is not big right now, but will be big then and not easily backwards-compatible since special-casing is too easy to over fit for specific applications.

How complex is the mode for open? Correct me if I'm wrong, but it is not a language of infinite size; can't it be handled simply by many @overloads in the stub file?

[sixolet]

Back when there were a couple interns working on this feature for a week, we had a few different use cases that they wanted to support:

• open (a finite string-based language)
• pow (wanted to dispatch on positive vs. negative exponents)
• subprocess.check_output

What else? Is pow an odd one out here? Is open? Or are these complicated ones rare, and in the common case you just want to dispatch on a constant?

[elazarg]

Maybe also Tuple * 5... Maybe pow can be handled by having positive and negative integer "subset types".

Any finite language is dispatch on a constant.

[JelleZijlstra]

As of 3.6, the subprocess.check_output case is actually more complicated: it returns str if universal_newlines is True OR encoding is non-None, and bytes otherwise. I suppose existing overloads can take care of the encoding part: you'd have overloads for encoding is non-None returning str, universal_newlines=True returning str, and universal_newlines=False returning bytes.

[gvanrossum]

But the main thing we really want is determining the correct return type when the argument is a compile-time literal. You can't overload on True or False because there's no way to spell "the subtype of bool that only contains True" (etc.). If it can't be determined what the value is at compile time it's okay to revert to the more general overload. Similar for open() and everything that takes a similar mode string (it's bred quite a following).

[pkch]

I would prefer this syntax:

from pathlib import Path

# type function
def open_return_type(kwargs: Dict[str, object]) -> Type[object]:
    assert isinstance(kwargs['mode'], str)
    if 'b' in kwargs['mode']:
        return IO[bytes]
    else:
        return IO[str]

def open(file: Union[str, bytes, int, Path], mode: str = 'r', buffering: int = -1,
         encoding: str = None, errors: str = None, newline: str = None,
         closefd: bool = ...) -> open_return_type: ...

It's flexible, and can be implemented like this:

• TypeAnalyser.visit_unbound_type needs to allow "type functions" in function annotation
• TypeChecker.check_return_statement, if it sees a type function, needs to call it, passing it the dict of all the arguments for which it knows literal values
• if the type function returns a value, the type checker can use it as the return type
• if the type function raises KeyError, the type checker should assume it relies on an argument that it doesn't know the value of; since it cannot infer the return type, it should do something sensible (e.g., issue a warning and use the return type based on the type context)

[elazarg]

if it sees a type function, needs to call it,

How? Running arbitrary user code is certainly out of question. What do you suggest? To me this sounds like a turing-complete metaprogramming language and full-blown dependent type system.

I do like is the idea of avoiding the scoping problem (how to express def f(x, y:g(x)): ... using another function. But it will only be needed for fully-dependent type system, not for singleton-types.

[pkch]

@elazarg Ah yes, I did mean to run arbitrary user code. If the type checker is not allowed to do that for security or other reasons, my suggestion won't work.

[JelleZijlstra]

It can't really do that in general, since there's nothing requiring type checkers to run in the same Python version as they're checking for, or even in Python at all.

[elazarg]

Just found this: https://typesandkinds.wordpress.com/#what-are-singletons