Refactor `std` to improve ease of porting

[jethrogb]

Currently, std contains platform-specific code throughout. This makes both
in-tree and out-of-tree ports harder. In-tree ports are harder because it's not
obvious what all needs to be implemented. Out-of-tree ports are harder because
they require frequent laborous rebasing due to an unclear interface. It's not
possible to port just a little bit to get your platform started, because
succesful compilation requires “everything” to be there.

The idea is to add a platform abstraction layer (PAL) to std to separate
out platform-specific code from generic code. The exact shape of the desired
interface is unclear. The refactoring will likely need to be an incremental
process where the PAL interface gets more cleary defined step-by-step.

As part of this work, it should be possible to create a "mock" platform that
can be used to test the interface and as a starting point for implementing new
platforms.

Prior work:

• One extreme ultimate goal was described by brson in 2016.
• Experimentation by @panicbit has shown that the current API can't really be captured in traits.
• Tracking issue for location of facade crates rust-lang/rust#27783: Tracking issue for the creates beneath the facade (bullet point added by @Ericson2314)

[panicbit]

Experimentation by @panicbit has shown that the current API can't really be captured in traits.

That is quite dated by now. I've tried a new approach which is a lot more straight forward and less idealistic. It will allow for a "mock" platform in the shape of the abstract_platform crate (which is more or less a copy of libstd but generic over the platform impl).

The current hurdles for a PAL in the form of traits are:

1. lack of GATs, which would allow for a sane abstraction of generic types
2. lack of statics and generic consts (can be worked around using functions and unlikely to be implemented in Rust)
3. An ICE related to traits and DST layouts (e.g. with Path)

[Ericson2314]

As I mentioned a bit on IRC, I would propose these 3 parallel first steps:

• (Proposed refactoring to implement core::io rust-lang/rfcs#2262) core::io, which is generally useful, and especially for the PAL which cannot access std

  1. core::io::Read and core::io::Write
     • "blanket impls" to derive std::io::{Read, Write} from core::io::{Read, Write}
  2. io::Error should be usable without allocation #12: core::io:Error solution
     • @panicbit proposed (on IRC) enum with prefix of variants of std::io::Error, so conversion should optimize away
     • I proposed (in io::Error should be usable without allocation #12 (comment)) no core::io::Error, but associated error type for core::io::{Read, Write} so as to better limit per-impl what errors are possible
  3. Associated "combinators types" like core::io::Chain and core::io::Cursor

• pure alloc crate (edit pulled out into A pure alloc library, with today's alloc becoming global_alloc #20)

• Move HashMap to liballoc rust-lang/rust#51846 alloc::Hashmap. That PR uses lang items. My original ideas for quick fix below:

  1. Just put up with the boilerplate of a manual newtype in std to get the ball rolling.
  2. Work with lang team on something to cut down on boilerplate. I propose permenantly-unstable auto-newtype feature (could maybe be viewed either as a special abstract type or struct wrapper).

(Besides this issue, I think we can also also get started on #5 immediately, and in total parallel to this.)

[jethrogb]

I actually think these are all orthogonal to this issue. The things you bring up make it easier to use things from std without using std. They don't make it easier to port the platform-dependent stuff of std.

[panicbit]

@jethrogb Well, the abstract_platform approach is all about being able to use things from std without using std, so the suggested changes would reduce the amount of boilerplate in the abstract_platform.

[Ericson2314]

@panicbit beat me to it :). HashMap probably isn't needed in the platform-dependent layers, but Read and Write traits are very useful.

[panicbit]

@Ericson2314 Indeed, the only platform-dependent use I could find was in a test in sys_common/net.rs

[Ericson2314]

As to fallible allocation, I can sort of squint at to make a concern for us. On certain platforms, fallible allocation is the only one that makes sense, so its sort of a policy-portability concern. Then, insofar that std, or as much of std as possible, ought to be ported to to those platforms, the platform-dependent layers should be able to use it.

I moved it below core::io::{Read, Write} so the 3 bullets are in order from most relevant to our goals to least relevant.

[jethrogb]

I see, these changes are necessary to deal with the dependency inversion problem.

[clarfonthey]

IMHO trait aliases make something like core::io possible; instead of hard-coding the error type you make it an associated type, and then make std::io versions trait aliases which explicitly set the error type.

[Ericson2314]

@clarcharr That would be really cool, but Read::read_to_end by accumulating a Vec makes its not possible: that method must be in a std or alloc trait. I think there'd also be more coherence/back-compat issues maybe? Like it would be cool to also make

impl<T: core::io::Write<Err = !>> core::hash::Hasher for T { .. }

which wouldn't work if both were trait aliases (but maybe it's if just one is!).

[Side note, in my previous attempt at this, I also changed some signatures to better reflect how some of the methods treat "end of file" as an error, and some return Ok(0): https://github.com/QuiltOS/core-io/blob/master/src/lib.rs#L112. Not sure whether we allow ourselves to "change" the method signatures like that, but if so that would also require a new trait of course.]

[clarfonthey]

@Ericson2314 That might be possible with some version of rust-lang/rfcs#2315, or if std is one crate and the method is marked with #[cfg(alloc)] or something similar.