proposal: change standard library to check for io.EOF using errors.Is

[tonistiigi]

Go1.13 introduced errors.Is/As/Unwrap for adding typed information to errors. With this change, errors turn into chains of values instead of a plain string. I think this is a significant improvement, but one of the issues stopping us from getting the full benefit from this is that the error checks inside stdlib still use the old == equality check instead of errors.Is.

This means that whenever we wrap errors with extra type information, we need to make sure these errors are not returned by any interface that might be passed to stdlib.

The most common case for this is io.Reader that should return io.EOF when there is no more data. With a quick search, I found that there are over 200 places in stdlib today where EOF is checked with == condition. For a typical example, io.Copy uses a check like this to determine if reader error also becomes a copy error. If we would change all these places to use errors.Is(err, io.EOF) instead, custom implementations of io.Reader could return much more useful errors.

For example, http.RoundTripper implementation could associate the request/response body with the information about the request, so that when an error happens later, it contains information about what specific request produced the data that ended up failing. Currently, most of the time, we would just get an unexpected EOF or broken pipe in this case.

I'm not suggesting any of the current io.EOF errors returned from stdlib (eg. os.File) should be wrapped. That would not be backward compatible. But just changing the internal equality checks and documenting it should be harmless.

Initially, 3rd party libraries will still likely check io.EOF directly, but over time they will get updated as well. Before stdlib makes the change and provides an official recommendation, they don't have any incentive to do that.

[ianlancetaylor]

I believe that io.EOF is the exception here, so retitling.

[ianlancetaylor]

The standard library says EOF is the error returned by Read when no more input is available.. I think we agree that that can't change without breaking the Go 1 guarantee. Since io.Copy is using Read, it seems OK that io.Copy checks using == io.EOF rather than errors.Is. The question is whether there are places in the standard library that use == io.EOF even though they are checking errors that are not returned from a Read method.

[tonistiigi]

I believe that io.EOF is the exception here, so retitling.

The issue I'm hitting is definitely with io.EOF but I haven't done a full scan. io.ErrUnexpectedEOF seems to have similar checks, although usually it is generated by stdlib itself (but doesn't need to be).

The standard library says EOF is the error returned by Read when no more input is available.

The semantic question in here is what does "is the error" mean after go1.13 . Is an error that is wrapped still the same error? From the method names (and design objectives) added in go1.13 it would strongly suggest that it is.

I think we agree that that can't change without breaking the Go 1 guarantee.

All the Read() methods in stdlib that generate io.EOF directly should continue to do that. Old code that used this and does io.EOF check with == will continue to work.

As I described, after this would ship, and a 3rd party library decides to wrap an IO error, they could break another 3rd party library that upgrades to use them. But 3rd party libraries are not under any go1 backward compatibility guarantees. Nobody can be sure they don't decide to not implement io.EOF at all. Semver can be used in these modules to signal changes.

It will definitely take time for all people to always use errors.Is when dealing with errors. My point is that the quicker we take some steps for transition, the sooner the messy period will be over. As this doesn't break anyone, it could be just considered following a best practice like the errors package suggests.

The question is whether there are places in the standard library that use == io.EOF even though they are checking errors that are not returned from a Read method.

My issue definitely is that there is no way to add typed information to errors that arise through io processing (at least without forking a significant portion of stdlib). If there are other cases they are less likely to actually solve the issues.

[ianlancetaylor]

The semantic question in here is what does "is the error" mean after go1.13 . Is an error that is wrapped still the same error? From the method names (and design objectives) added in go1.13 it would strongly suggest that it is.

Personally I don't think we have that leeway. I think that the contract of the standard and widely-used Read method is that it should return exactly io.EOF on end-of-file.

That said, I think you are suggesting that where the standard library calls Read, it should check for io.EOF using errors.Is rather than ==. I know you've mentioned other errors too, but I think we have to be specific about what is going to change here. Without looking at real code we don't know what other programs might break. For example, you are suggesting that we change io.ReadAtLeast to use errors.Is(err, EOF) rather than err == EOF.

I don't think that would do any real harm but I'm not personally convinced that it is necessary, given my reading of how I think Read is required to behave.

Happy to hear other opinions.

[tiborvass]

I'm curious to understand how adjusting the docs about io.EOF would break compatibility between two 3rd-party codebases in a way that's not already possible today. Furthermore, if the error checking paradigm from Go 1.13 is fully embraced, shouldn't tooling such as linters or maybe even gofix also prefer errors.Is over equality?

[dsnet]

Unless we change all code to use errors.Is on io.EOF instead of performing a direct == comparison (which we cannot feasibly do), I believe having the standard library use errors.Is is more harmful than good. It provides the illusion that wrapping io.EOF in Read is permissible, when it is not. Some users will depend on that behavior and think that its okay to return wrapped io.EOF errors since it works with the standard library, but be surprised that it doesn't work with some other library that uses io.Reader, despite that library being fully compliant with the current specified behavior.

I'm curious to understand how adjusting the docs about io.EOF would break compatibility between two 3rd-party codebases in a way that's not already possible today. Furthermore, if the error checking paradigm from Go 1.13 is fully embraced, shouldn't tooling such as linters or maybe even gofix also prefer errors.Is over equality?

It's not an issue of whether tooling can migrate existing code. It's a question of whether correct code today will continue to function after the documented change. Most code today only do an err == io.EOF check. Changing the documentation makes it such that this correct code is now incorrect by redefinition.

[SamWhited]

TL;DR — I have a general rule of thumb that I don't return io.EOF and always return a new error if I need more context, or, more frequently, handle the io.EOF at the read call site because EOF isn't actually an error. We should not encourage checking that wrapped errors started out in life as an EOF, we should treat them like any other error.

---

I tend to think of io.EOF as a special case; a sentinel value indicating expected behavior, not an error. This already leads to the unfortunate "if err != nil && err != io.EOF" scattered throughout my codebase anywhere some code calls Read. It also frequently leads to bugs in my code where I either forget to check for and handle the EOF from a Read call, or I some piece of code returns EOF when I'm not expecting it and I can't always easily track down where such a generic error message is coming from. This all happens because EOF is not an error, it's expected behavior. The error comes later when I go to do something with my data and realize that the expected data wasn't completely read, or was the wrong data, etc.

Because of this, I don't think I've ever seen a place where checking for io.EOF further up the stack (not directly at the call site) would do anything but hide a bug where I wanted to handle io.EOF before, and I don't think this should be encouraged. There are occasions where we want to actually return an io.EOF (eg. if the function in question also has read like semantics), so we need a way to distinguish between an actual new error and an EOF while still keeping context around. EOF doesn't provide any context, and one read site that returned EOF is as good as another, so I generally don't think it should be wrapped with more context if we want to continue to treat it as a magical sentinel value.

This means that if I really want to check for io.EOF further up the stack, I should just return it and not wrap it so that it can be treated as a signal, not an error. If I do want to signal to a user of my code that the EOF was unexpected and therefore an actual error, I should return an error, not EOF (which does not indicate that an error occurred). If I'm a user of a library that has wrapped EOF, this to me indicates that this was an actual error and I should treat the wrapped error just like I would any other error (and it doesn't matter that the root error was an EOF signal).

[as]

I tend to think of io.EOF as a special case; a sentinel value indicating expected behavior, not an error.

This is only true if the "file" is not infinite. It is the user/developer that has the expectation that there will ever be an end-of-file, not the io.Reader interface. I see it as an expected error rather than a special case, hence why the concrete value is so ubiquitous.