Rollup merge of #142134 - workingjubilee:reject-unsupported-abi, r=jdonszelmann,RalfJung

Reject unsupported `extern "{abi}"`s consistently in all positions

Modify the handling of `extern "{abi}"` in the compiler so that it has consistent errors without regard to the position in the grammar.

## What

Implement the following breakages:
- Promote [`unsupported_fn_ptr_calling_conventions`] from a warning to a hard error
- Guarantee future edge-cases like trait declarations will not escape so that *ABIs that have already been unusable in most positions* will now be unusable in **all positions**. See "How" and "Why or Why Not" for more details.

In particular, these architecture-specific ABIs now only compile on their architectures[^4]:
  - amdgpu: "gpu-kernel"
  - arm: "aapcs", "C-cmse-nonsecure-entry", "C-cmse-nonsecure-call"
  - avr: "avr-interrupt", "avr-non-blocking-interrupt"
  - msp430: "msp430-interrupt"
  - nvptx64: "gpu-kernel", "ptx-kernel"
  - riscv32 and riscv64: "riscv-interrupt-m", "riscv-interrupt-s"
  - x86: "thiscall"
  - x86 and x86_64: "x86-interrupt"
  - x86_64: "sysv64", "win64"

ABIs that are logically x86-specific but actually permitted on all Windows targets remain permitted on Windows, as before. For non-Windows targets, they error if we had previously done so in other positions.

## How

We modify rustc_ast_lowering to prevent unsupported ABIs from leaking through the HIR without being checked for target support. They now emit hard errors for every case where we would return `Invalid` from `AbiMap::canonize_abi`. Previously ad-hoc checking on various HIR items required making sure we check every HIR item which could contain an `extern "{abi}"` string. This is a losing proposition compared to gating the lowering itself.

As a consequence, unsupported ABI strings error instead of triggering the warning `unsupported_fn_ptr_calling_conventions`. The code is also simpler compared to alternative implementations that might e.g. split on unstable vs. stable, only suffering some unavoidable complication to support the newly-revived `unsupported_calling_conventions` lint.[^5]

However, per #86232 this does cause errors for rare usages of `extern "{abi}"` that were theoretically possible to write in Rust source, without previous warning or error. For instance, trait declarations without impls were never checked. These are the exact kinds of leakages that this new approach prevents.

This differs from the following PRs:
- #141435 is orthogonal, as it adds a new lint for ABIs we have not warned on and are not touched by this PR
- #141877 is subsumed by this, in that this simply cuts out bad functionality instead of adding epicycles for stable code

## Why or Why Not

We already made the decision to issue the `unsupported_fn_ptr_calling_conventions` future compatibility warning. It has warned in dependencies since #135767, which reached stable with Rust 1.87. That was released on 2025 May 17, and it is now June. As we already had erred on these ABI strings in most other positions, and warn on stable for function pointer types, this breakage has had reasonable foreshadowing.

Upgrading the warning to an error addresses a real problem. In some cases the Rust compiler can attempt to actually compute the ABI for calling a function with an unsupported ABI. We could accept this case and compute unsupported ABIs according to some other ABI, silently[^6]. However, this obviously exposes Rust to errors in codegen. We cannot lower directly to the "obvious", target-incorrect ABI and then trust code generators like LLVM to reliably error on these cases, either.

Other considerations include:
- We could refactor the compiler to defer ABI computations, but that seems like it would suffer the "whack-a-mole" problem close to linking instead of after parsing and expansion.
- We could run a deprecation cycle for the edge cases, but we would be warning highly marginal cases, like this trait declaration without a definition that cannot be implemented without error[^9].
```rust
pub trait UsedToSneakBy {
    pub extern "gpu-kernel" fn sneaky();
}
```
- The [crater run] on this PR's draft form suggests the primary issue is with implementations on function pointers, which has already been warned on, so it does not seem like we would be benefiting any real code.
- Converting this to a hard error now, in the same cycle that we ship the reanimation of the `unsupported_calling_conventions` lint, means people who would otherwise have to deal with two lints only have to update their code in one batch. Of course, one of them is as breakage.

r? lang

Fixes #86232
Fixes #132430
Fixes #138738
Fixes #142107

[`unsupported_fn_ptr_calling_conventions`]: #130260
[crater run]: #142134 (comment)
[^9]: Upon any impl, even for provided fn within trait declarations, e.g. `pub extern "gpu-kernel" fn sneaky() {}`, different HIR types were used which would, in fact, get checked. Likewise for anything with function pointers. Thus we would be discussing deprecation cycles for code that is impotent or forewarned[^7].
[^4]: Some already will not compile, due to reaching ICEs or LLVM errors.
[^5]: That lint cannot be moved in a similar way yet because lints operate on HIR, so you cannot emit lints when the HIR has not been completely formed.
[^6]:  We already do this for all `AbiStr` we cannot parse, pretending they are `ExternAbi::Rust`, but we also emit an error to prevent reaching too far into codegen.
[^7]: It actually did appear in two cases in rustc's test suite because we are a collection of Rust edge-cases by the simple fact that we don't care if the code actually runs. These cases are being excised in 643a9d2

Author: workingjubilee

compiler/rustc_ast_lowering/src/item.rs

@@ -2,7 +2,7 @@ use rustc_abi::ExternAbi;
 use rustc_ast::ptr::P;
 use rustc_ast::visit::AssocCtxt;
 use rustc_ast::*;
-use rustc_errors::ErrorGuaranteed;
+use rustc_errors::{E0570, ErrorGuaranteed, struct_span_code_err};
 use rustc_hir::def::{DefKind, PerNS, Res};
 use rustc_hir::def_id::{CRATE_DEF_ID, LocalDefId};
 use rustc_hir::{self as hir, HirId, LifetimeSource, PredicateOrigin};
@@ -1644,9 +1644,29 @@ impl<'hir> LoweringContext<'_, 'hir> {
             self.error_on_invalid_abi(abi_str);
             ExternAbi::Rust
         });
-        let sess = self.tcx.sess;
-        let features = self.tcx.features();
-        gate_unstable_abi(sess, features, span, extern_abi);
+        let tcx = self.tcx;
+
+        // we can't do codegen for unsupported ABIs, so error now so we won't get farther
+        if !tcx.sess.target.is_abi_supported(extern_abi) {
+            let mut err = struct_span_code_err!(
+                tcx.dcx(),
+                span,
+                E0570,
+                "{extern_abi} is not a supported ABI for the current target",
+            );
+
+            if let ExternAbi::Stdcall { unwind } = extern_abi {
+                let c_abi = ExternAbi::C { unwind };
+                let system_abi = ExternAbi::System { unwind };
+                err.help(format!("if you need `extern {extern_abi}` on win32 and `extern {c_abi}` everywhere else, \
+                    use `extern {system_abi}`"
+                ));
+            }
+            err.emit();
+        }
+        // Show required feature gate even if we already errored, as the user is likely to build the code
+        // for the actually intended target next and then they will need the feature gate.
+        gate_unstable_abi(tcx.sess, tcx.features(), span, extern_abi);
         extern_abi
     }
 

compiler/rustc_hir_analysis/src/check/check.rs

@@ -1,7 +1,7 @@
 use std::cell::LazyCell;
 use std::ops::ControlFlow;
 
-use rustc_abi::FieldIdx;
+use rustc_abi::{ExternAbi, FieldIdx};
 use rustc_attr_data_structures::ReprAttr::ReprPacked;
 use rustc_attr_data_structures::{AttributeKind, find_attr};
 use rustc_data_structures::unord::{UnordMap, UnordSet};
@@ -13,7 +13,6 @@ use rustc_infer::infer::{RegionVariableOrigin, TyCtxtInferExt};
 use rustc_infer::traits::{Obligation, ObligationCauseCode};
 use rustc_lint_defs::builtin::{
     REPR_TRANSPARENT_EXTERNAL_PRIVATE_FIELDS, UNSUPPORTED_CALLING_CONVENTIONS,
-    UNSUPPORTED_FN_PTR_CALLING_CONVENTIONS,
 };
 use rustc_middle::hir::nested_filter;
 use rustc_middle::middle::resolve_bound_vars::ResolvedArg;
@@ -53,49 +52,22 @@ fn add_abi_diag_help<T: EmissionGuarantee>(abi: ExternAbi, diag: &mut Diag<'_, T
 }
 
 pub fn check_abi(tcx: TyCtxt<'_>, hir_id: hir::HirId, span: Span, abi: ExternAbi) {
-    // FIXME: this should be checked earlier, e.g. in `rustc_ast_lowering`, to fix
-    // things like #86232.
+    // FIXME: This should be checked earlier, e.g. in `rustc_ast_lowering`, as this
+    // currently only guards function imports, function definitions, and function pointer types.
+    // Functions in trait declarations can still use "deprecated" ABIs without any warning.
 
     match AbiMap::from_target(&tcx.sess.target).canonize_abi(abi, false) {
         AbiMapping::Direct(..) => (),
+        // already erred in rustc_ast_lowering
         AbiMapping::Invalid => {
-            let mut err = struct_span_code_err!(
-                tcx.dcx(),
-                span,
-                E0570,
-                "`{abi}` is not a supported ABI for the current target",
-            );
-            add_abi_diag_help(abi, &mut err);
-            err.emit();
+            tcx.dcx().span_delayed_bug(span, format!("{abi} should be rejected in ast_lowering"));
         }
         AbiMapping::Deprecated(..) => {
             tcx.node_span_lint(UNSUPPORTED_CALLING_CONVENTIONS, hir_id, span, |lint| {
-                lint.primary_message("use of calling convention not supported on this target");
-                add_abi_diag_help(abi, lint);
-            });
-        }
-    }
-}
-
-pub fn check_abi_fn_ptr(tcx: TyCtxt<'_>, hir_id: hir::HirId, span: Span, abi: ExternAbi) {
-    // This is always an FCW, even for `AbiMapping::Invalid`, since we started linting later than
-    // in `check_abi` above.
-    match AbiMap::from_target(&tcx.sess.target).canonize_abi(abi, false) {
-        AbiMapping::Direct(..) => (),
-        // This is not a redundant match arm: these ABIs started linting after introducing
-        // UNSUPPORTED_FN_PTR_CALLING_CONVENTIONS already existed and we want to
-        // avoid expanding the scope of that lint so it can move to a hard error sooner.
-        AbiMapping::Deprecated(..) => {
-            tcx.node_span_lint(UNSUPPORTED_CALLING_CONVENTIONS, hir_id, span, |lint| {
-                lint.primary_message("use of calling convention not supported on this target");
-                add_abi_diag_help(abi, lint);
-            });
-        }
-        AbiMapping::Invalid => {
-            tcx.node_span_lint(UNSUPPORTED_FN_PTR_CALLING_CONVENTIONS, hir_id, span, |lint| {
                 lint.primary_message(format!(
-                    "the calling convention {abi} is not supported on this target"
+                    "{abi} is not a supported ABI for the current target"
                 ));
+                add_abi_diag_help(abi, lint);
             });
         }
     }
@@ -868,6 +840,7 @@ pub(crate) fn check_item_type(tcx: TyCtxt<'_>, def_id: LocalDefId) {
             let hir::ItemKind::ForeignMod { abi, items } = it.kind else {
                 return;
             };
+
             check_abi(tcx, it.hir_id(), it.span, abi);
 
             for item in items {

compiler/rustc_hir_analysis/src/check/mod.rs

@@ -72,8 +72,8 @@ pub mod wfcheck;
 
 use std::num::NonZero;
 
-pub use check::{check_abi, check_abi_fn_ptr, check_custom_abi};
-use rustc_abi::{ExternAbi, VariantIdx};
+pub use check::{check_abi, check_custom_abi};
+use rustc_abi::VariantIdx;
 use rustc_data_structures::fx::{FxHashSet, FxIndexMap};
 use rustc_errors::{Diag, ErrorGuaranteed, pluralize, struct_span_code_err};
 use rustc_hir::LangItem;

compiler/rustc_hir_analysis/src/hir_ty_lowering/mod.rs

@@ -51,7 +51,7 @@ use rustc_trait_selection::traits::wf::object_region_bounds;
 use rustc_trait_selection::traits::{self, FulfillmentError};
 use tracing::{debug, instrument};
 
-use crate::check::check_abi_fn_ptr;
+use crate::check::check_abi;
 use crate::errors::{AmbiguousLifetimeBound, BadReturnTypeNotation};
 use crate::hir_ty_lowering::errors::{GenericsArgsErrExtend, prohibit_assoc_item_constraint};
 use crate::hir_ty_lowering::generics::{check_generic_arg_count, lower_generic_args};
@@ -2660,7 +2660,7 @@ impl<'tcx> dyn HirTyLowerer<'tcx> + '_ {
         if let hir::Node::Ty(hir::Ty { kind: hir::TyKind::BareFn(bare_fn_ty), span, .. }) =
             tcx.hir_node(hir_id)
         {
-            check_abi_fn_ptr(tcx, hir_id, *span, bare_fn_ty.abi);
+            check_abi(tcx, hir_id, *span, bare_fn_ty.abi);
         }
 
         // reject function types that violate cmse ABI requirements

compiler/rustc_lint/src/lib.rs

@@ -608,6 +608,7 @@ fn register_builtins(store: &mut LintStore) {
         "converted into hard error, see issue #127323 \
          <https://github.com/rust-lang/rust/issues/127323> for more information",
     );
+    store.register_removed("unsupported_fn_ptr_calling_conventions", "converted into hard error");
     store.register_removed(
         "undefined_naked_function_abi",
         "converted into hard error, see PR #139001 \

compiler/rustc_lint_defs/src/builtin.rs

@@ -122,7 +122,6 @@ declare_lint_pass! {
         UNSAFE_OP_IN_UNSAFE_FN,
         UNSTABLE_NAME_COLLISIONS,
         UNSTABLE_SYNTAX_PRE_EXPANSION,
-        UNSUPPORTED_FN_PTR_CALLING_CONVENTIONS,
         UNUSED_ASSIGNMENTS,
         UNUSED_ASSOCIATED_TYPE_BOUNDS,
         UNUSED_ATTRIBUTES,

tests/auxiliary/minicore.rs

@@ -16,6 +16,7 @@
 
 #![feature(
     no_core,
+    intrinsics,
     lang_items,
     auto_traits,
     freeze_impls,
@@ -196,3 +197,9 @@ impl<'a, 'b: 'a, T: PointeeSized + Unsize<U>, U: PointeeSized> CoerceUnsized<&'a
 trait Drop {
     fn drop(&mut self);
 }
+
+pub mod mem {
+    #[rustc_nounwind]
+    #[rustc_intrinsic]
+    pub unsafe fn transmute<Src, Dst>(src: Src) -> Dst;
+}

tests/crashes/132430.rs

@@ -1,132 +1,75 @@
-warning: the calling convention "msp430-interrupt" is not supported on this target
-  --> $DIR/cannot-be-called.rs:60:18
+error[E0570]: "msp430-interrupt" is not a supported ABI for the current target
+  --> $DIR/cannot-be-called.rs:38:8
    |
-LL | fn msp430_ptr(f: extern "msp430-interrupt" fn()) {
-   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-   |
-   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
-   = note: for more information, see issue #130260 <https://github.com/rust-lang/rust/issues/130260>
-   = note: `#[warn(unsupported_fn_ptr_calling_conventions)]` on by default
+LL | extern "msp430-interrupt" fn msp430() {}
+   |        ^^^^^^^^^^^^^^^^^^
 
-warning: the calling convention "riscv-interrupt-m" is not supported on this target
-  --> $DIR/cannot-be-called.rs:74:19
-   |
-LL | fn riscv_m_ptr(f: extern "riscv-interrupt-m" fn()) {
-   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+error[E0570]: "riscv-interrupt-m" is not a supported ABI for the current target
+  --> $DIR/cannot-be-called.rs:42:8
    |
-   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
-   = note: for more information, see issue #130260 <https://github.com/rust-lang/rust/issues/130260>
+LL | extern "riscv-interrupt-m" fn riscv_m() {}
+   |        ^^^^^^^^^^^^^^^^^^^
 
-warning: the calling convention "riscv-interrupt-s" is not supported on this target
-  --> $DIR/cannot-be-called.rs:81:19
-   |
-LL | fn riscv_s_ptr(f: extern "riscv-interrupt-s" fn()) {
-   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+error[E0570]: "riscv-interrupt-s" is not a supported ABI for the current target
+  --> $DIR/cannot-be-called.rs:44:8
    |
-   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
-   = note: for more information, see issue #130260 <https://github.com/rust-lang/rust/issues/130260>
+LL | extern "riscv-interrupt-s" fn riscv_s() {}
+   |        ^^^^^^^^^^^^^^^^^^^
 
-warning: the calling convention "x86-interrupt" is not supported on this target
-  --> $DIR/cannot-be-called.rs:88:15
+error[E0570]: "x86-interrupt" is not a supported ABI for the current target
+  --> $DIR/cannot-be-called.rs:46:8
    |
-LL | fn x86_ptr(f: extern "x86-interrupt" fn()) {
-   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
-   |
-   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
-   = note: for more information, see issue #130260 <https://github.com/rust-lang/rust/issues/130260>
+LL | extern "x86-interrupt" fn x86() {}
+   |        ^^^^^^^^^^^^^^^
 
-error[E0570]: `"msp430-interrupt"` is not a supported ABI for the current target
-  --> $DIR/cannot-be-called.rs:36:1
+error[E0570]: "msp430-interrupt" is not a supported ABI for the current target
+  --> $DIR/cannot-be-called.rs:65:25
    |
-LL | extern "msp430-interrupt" fn msp430() {}
-   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+LL | fn msp430_ptr(f: extern "msp430-interrupt" fn()) {
+   |                         ^^^^^^^^^^^^^^^^^^
 
-error[E0570]: `"riscv-interrupt-m"` is not a supported ABI for the current target
-  --> $DIR/cannot-be-called.rs:40:1
+error[E0570]: "riscv-interrupt-m" is not a supported ABI for the current target
+  --> $DIR/cannot-be-called.rs:77:26
    |
-LL | extern "riscv-interrupt-m" fn riscv_m() {}
-   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+LL | fn riscv_m_ptr(f: extern "riscv-interrupt-m" fn()) {
+   |                          ^^^^^^^^^^^^^^^^^^^
 
-error[E0570]: `"riscv-interrupt-s"` is not a supported ABI for the current target
-  --> $DIR/cannot-be-called.rs:42:1
+error[E0570]: "riscv-interrupt-s" is not a supported ABI for the current target
+  --> $DIR/cannot-be-called.rs:83:26
    |
-LL | extern "riscv-interrupt-s" fn riscv_s() {}
-   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+LL | fn riscv_s_ptr(f: extern "riscv-interrupt-s" fn()) {
+   |                          ^^^^^^^^^^^^^^^^^^^
 
-error[E0570]: `"x86-interrupt"` is not a supported ABI for the current target
-  --> $DIR/cannot-be-called.rs:44:1
+error[E0570]: "x86-interrupt" is not a supported ABI for the current target
+  --> $DIR/cannot-be-called.rs:89:22
    |
-LL | extern "x86-interrupt" fn x86() {}
-   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+LL | fn x86_ptr(f: extern "x86-interrupt" fn()) {
+   |                      ^^^^^^^^^^^^^^^
 
 error: functions with the "avr-interrupt" ABI cannot be called
-  --> $DIR/cannot-be-called.rs:50:5
+  --> $DIR/cannot-be-called.rs:53:5
    |
 LL |     avr();
    |     ^^^^^
    |
 note: an `extern "avr-interrupt"` function can only be called using inline assembly
-  --> $DIR/cannot-be-called.rs:50:5
+  --> $DIR/cannot-be-called.rs:53:5
    |
 LL |     avr();
    |     ^^^^^
 
 error: functions with the "avr-interrupt" ABI cannot be called
-  --> $DIR/cannot-be-called.rs:70:5
+  --> $DIR/cannot-be-called.rs:73:5
    |
 LL |     f()
    |     ^^^
    |
 note: an `extern "avr-interrupt"` function can only be called using inline assembly
-  --> $DIR/cannot-be-called.rs:70:5
+  --> $DIR/cannot-be-called.rs:73:5
    |
 LL |     f()
    |     ^^^
 
-error: aborting due to 6 previous errors; 4 warnings emitted
+error: aborting due to 10 previous errors
 
 For more information about this error, try `rustc --explain E0570`.
-Future incompatibility report: Future breakage diagnostic:
-warning: the calling convention "msp430-interrupt" is not supported on this target
-  --> $DIR/cannot-be-called.rs:60:18
-   |
-LL | fn msp430_ptr(f: extern "msp430-interrupt" fn()) {
-   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-   |
-   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
-   = note: for more information, see issue #130260 <https://github.com/rust-lang/rust/issues/130260>
-   = note: `#[warn(unsupported_fn_ptr_calling_conventions)]` on by default
-
-Future breakage diagnostic:
-warning: the calling convention "riscv-interrupt-m" is not supported on this target
-  --> $DIR/cannot-be-called.rs:74:19
-   |
-LL | fn riscv_m_ptr(f: extern "riscv-interrupt-m" fn()) {
-   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-   |
-   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
-   = note: for more information, see issue #130260 <https://github.com/rust-lang/rust/issues/130260>
-   = note: `#[warn(unsupported_fn_ptr_calling_conventions)]` on by default
-
-Future breakage diagnostic:
-warning: the calling convention "riscv-interrupt-s" is not supported on this target
-  --> $DIR/cannot-be-called.rs:81:19
-   |
-LL | fn riscv_s_ptr(f: extern "riscv-interrupt-s" fn()) {
-   |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-   |
-   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
-   = note: for more information, see issue #130260 <https://github.com/rust-lang/rust/issues/130260>
-   = note: `#[warn(unsupported_fn_ptr_calling_conventions)]` on by default
-
-Future breakage diagnostic:
-warning: the calling convention "x86-interrupt" is not supported on this target
-  --> $DIR/cannot-be-called.rs:88:15
-   |
-LL | fn x86_ptr(f: extern "x86-interrupt" fn()) {
-   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
-   |
-   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
-   = note: for more information, see issue #130260 <https://github.com/rust-lang/rust/issues/130260>
-   = note: `#[warn(unsupported_fn_ptr_calling_conventions)]` on by default
-