Report allocation errors as panics, second attempt

[Amanieu]

Attempt to re-land #109507 now that #110771 is fixed.

---

OOM is now reported as a panic but with a custom payload type (AllocErrorPanicPayload) which holds the layout that was passed to handle_alloc_error.

This should be reviewed one commit at a time:

• The first commit adds AllocErrorPanicPayload and changes allocation errors to always be reported as panics.
• The second commit removes #[alloc_error_handler] and the alloc_error_hook API.

ACP: rust-lang/libs-team#192

Closes #51540
Closes #51245

[rustbot]

r? @jackh726

(rustbot has picked a reviewer for you, use r? to override)

[rustbot]

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo

[bors]

☔ The latest upstream changes (presumably #112671) made this pull request unmergeable. Please resolve the merge conflicts.

[jackh726]

Should this be assigned to someone on libs team? I guess it's mostly compiler work?

[Amanieu]

Yes, most of the compiler changes are just removing #[alloc_error_handler].

r? libs

[breathx]

@Amanieu could you say if some updates on it planned? Why not merging into master?

[Amanieu]

It's waiting on review.

[bors]

☔ The latest upstream changes (presumably #113014) made this pull request unmergeable. Please resolve the merge conflicts.

[Amanieu]

r? libs-api

[bors]

☔ The latest upstream changes (presumably #113162) made this pull request unmergeable. Please resolve the merge conflicts.

[bjorn3]

This one is still necessary.

[Amanieu]

Oops, that was a mistake.

[bors]

☔ The latest upstream changes (presumably #113391) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #116578) made this pull request unmergeable. Please resolve the merge conflicts.

[RalfJung]

The issue at swc-project/swc#8362 and following discussion on Zulip uncovered some subtle concerns around allocation-fail-handlers -- and this here is the first time user-defined code gets run on allocation failure. (Specifically, that user-defined code is the panic hook.)

The problem in that issue is that the library assumes there is no reentrancy, but it does cause an allocation, and so if that allocation fails and then the panic handler calls back into the library, we have an unsoundness due to aliasing references (or due to any of the other problems that reentrancy can cause). We have to globally decide that either libraries must generally assume that calling the allocator will run arbitrary safe code (and add safeguards against reentrancy where needed), or the code that runs on allocation failure needs to be "careful" to not touch state that might be invalid since the library governing that state is in the middle of something.

[Dylan-DPC]

This has conflicts to be resolved before getting reviewed

[bjorn3]

and this here is the first time user-defined code gets run on allocation failure. (Specifically, that user-defined code is the panic hook.)

We already have -Zoom=panic when libstd is used. This only additionally runs user code on no_std platforms. Also for an allocation failure to happen, you did generally first try to allocate something, which already runs user code. I don't think it is likely code would be unsound when user code runs on allocation failure, but not when user code runs on allocation itself.

[RalfJung]

Also for an allocation failure to happen, you did generally first try to allocate something, which already runs user code.

Global allocators are unsafe, so if that code does anything funky, we can blame that code.

I don't think it is likely code would be unsound when user code runs on allocation failure, but not when user code runs on allocation itself.

swc-project/swc#8362 is an example of that.

It seems entirely reasonable to say that the allocator must not call back into an AtomStoreCell in that way. But we cannot stop the panic handler from doing that.

[bjorn3]

Actually, turns out for no_std applications we already invoke the panic handler when #[rust_panic_handler] is not defined:

rust/library/alloc/src/alloc.rs

Lines 438 to 441 in 8f21a5c

	core::panicking::panic_nounwind_fmt(
	format_args!("memory allocation of {size} bytes failed"),
	/* force_no_backtrace */ false,
	)

For example

#![no_std]
#![no_main]

extern crate alloc;

use core::alloc::{GlobalAlloc, Layout};

struct MyAlloc;

unsafe impl GlobalAlloc for MyAlloc {
    unsafe fn alloc(&self, _: Layout) -> *mut u8 { core::ptr::null_mut() }
    unsafe fn dealloc(&self, _: *mut u8, _: Layout) { todo!() }
}

#[global_allocator]
static ALLOC: MyAlloc = MyAlloc;

#[link(name = "c")]
extern "C" {}

#[panic_handler]
fn panic_handler(_: &core::panic::PanicInfo) -> ! { loop {} }

// This depends on an internal implementation detail, but is only necessary because I'm compiling
// for a target which defaults to panic=unwind.
#[unsafe(no_mangle)]
fn rust_eh_personality() -> ! { panic!() }

#[unsafe(no_mangle)]
fn main() {
    alloc::boxed::Box::new(1);
}

when compiled with panic=abort works just fine on stable and hangs inside the panic handler with the following backtrace:

#0  alloc_error_handler::panic_handler () at alloc_error_handler.rs:22
#1  0x000055555555518d in core::panicking::panic_nounwind_fmt::runtime () at library/core/src/panicking.rs:117
#2  core::panicking::panic_nounwind_fmt () at library/core/src/intrinsics/mod.rs:3241
#3  0x00005555555550d9 in alloc::alloc::__alloc_error_handler::__rdl_oom () at library/alloc/src/alloc.rs:437
#4  0x0000555555555063 in alloc::alloc::handle_alloc_error::rt_error () at library/alloc/src/alloc.rs:405
#5  alloc::alloc::handle_alloc_error () at library/alloc/src/alloc.rs:411
#6  0x00005555555557e3 in alloc::alloc::exchange_malloc (size=4, align=4)
    at /home/bjorn/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:352
#7  0x0000555555555c39 in alloc::boxed::{impl#0}::new<i32> (x=1) at /home/bjorn/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:261
#8  alloc_error_handler::main () at alloc_error_handler.rs:31

In other words this PR doesn't open any soundness holes that don't already exist.

[bjorn3]

This was originally implemented in #76448 and stabilized in #102318.

[RalfJung]

Yeah, no_std already has a problem. (Also see rust-lang/unsafe-code-guidelines#506.) But no_std binaries are subject to all sorts of subtle issues, so I don't think this is sufficient justification for making this a much bigger soundness problem that applies to all binaries.

[bjorn3]

I don't see any reason why we should handle the std and no_std differently here.

But no_std binaries are subject to all sorts of subtle issues

Like what?

[RalfJung]

You need to set up the entry point, and the unwinding personality, and generally you're (unsafely) interfacing with the OS loader (or whatever else loads your binary).

I don't see any reason why we should handle the std and no_std differently here.

A soundness bug that affects only a small amount of users is not the same as one that affects a much bigger set of users. no_std binaries are pretty niche.

I also just fundamentally, and very strongly, disagree with arguments of the sort "this is already unsound so it doesn't matter if we have more of it". By that standard we can just stop caring about soundness entirely.

So, a hard no from my side for exposing these kinds of soundness issues any further until we have made a decision for who's responsibility it is to deal with this reentrancy.

[bjorn3]

You need to set up the entry point, and the unwinding personality, and generally you're (unsafely) interfacing with the OS loader (or whatever else loads your binary).

The personality only applies when the standard library itself is compiled with panic=unwind. For a proper panic=abort target you don't need that, nor when you use -Zbuild-std. As for the entry point and OS loader, if you don't do something custom made, but instead are writing eg a UEFI executable, you only need a couple lines of unsafe code that could be fully encapsulated in a crate like the uefi crate (and in fact I would say this is actually more robust that the way libstd is implemented).

I also just fundamentally, and very strongly, disagree with arguments of the sort "this is already unsound so it doesn't matter if we have more of it". By that standard we can just stop caring about soundness entirely.

So, a hard no from my side for exposing these kinds of soundness issues any further until we have made a decision for who's responsibility it is to deal with this reentrancy.

I would argue that it isn't unsound to run user code in the alloc error handler, but rather that it is unsound to depend on the alloc error handler not running user code. Simply because there is no way to fix the no_std case to not run user code in the alloc error handler without either breaking literally every no_std (which would IMO be completely unacceptable) + alloc project in existence or to replace it with an abort without running the panic handler (which is unacceptable for a non-trivial amount of use cases and isn't possible on every architecture anyway) And having us say that no_std is permanently unsound wouldn't be great either.

As I see it, the stabilization in #102318 forced our hand in deciding who is responsible for this reentrancy.

[RalfJung]

It seems quite bad to say that it is impossible to make crates like https://github.com/swc-project/swc sound -- that'd permanently close the door for useful APIs that we should enable people to use. In particular, having that crate be sound seems a lot more useful than allowing the allocation failure handler to re-enter the crate it was invoked by.

One can imagine various transition strategies to achieve that, for instance:

• making #[panic_handler] unsafe in a future edition, with FCW in older editions
• calling some other, unsafe #[alloc_fail_handler] on alloc failure (and have it default e.g. to intrinsics::abort())

I don't think we have reached the point yet where we should just give up.