rustc build artifacts should contain checksum to pervent unrelated ICEs / miscompiles / etc

[klensy]

Checked with incremental compilation artifacts:

1. Created default cargo project with:
   main.rs:

fn main(){}

2. in Cargo.toml turned on incremental = true, codegen-units=1
3. cargo build
4. in target/debug/incremental/[some_path] folder opened some dep-graph.bin, query-cache.bin (didn't checked other types yet) and randomly changed few bytes.
5. opened main.rs and added\removed spaces around (to use incremental build)
6. cargo build

I'm expected, that build will be failed, because of wrong checksum/hash of corrupted files, and some error like: "files was corrupted, etc..",
but it actually builds successfully or ICEs with general message, like that (here i changed dep-graph file, i guess) (can be differ from time to time, so this exact ICE is not important)

thread 'rustc' panicked at 'called `Option::unwrap()` on a `None` value', compiler\rustc_metadata\src\rmeta\def_path_hash_map.rs:18:85
stack backtrace:
   0:      0x7fed7737e30 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he376fbea1245f2cf
   1:      0x7fed7767a9a - core::fmt::write::h5e138f42fa148c9f
   2:      0x7fed7729059 - <std::io::IoSlice as core::fmt::Debug>::fmt::h2ac239f531b51c25
   3:      0x7fed773b5a2 - std::panicking::default_hook::hdac2f70bb79c9285
   4:      0x7fed773b163 - std::panicking::default_hook::hdac2f70bb79c9285
   5:      0x7feb1798149 - <rustc_driver[d0f8e8bee9276d3d]::args::Error as core[caf1bcf4c9c99831]::fmt::Debug>::fmt
   6:      0x7fed773bed2 - std::panicking::rust_panic_with_hook::hf60975f0d343d471
   7:      0x7fed773bbe2 - <std::panicking::begin_panic_handler::StrPanicPayload as core::panic::BoxMeUp>::get::ha1e40f14bea3163b
   8:      0x7fed7738787 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he376fbea1245f2cf
   9:      0x7fed773b899 - rust_begin_unwind
  10:      0x7fed779ac95 - core::panicking::panic_fmt::h169c5b4c17c0b46c
  11:      0x7fed779ab3c - core::panicking::panic::h774c6a2306362c68
  12:      0x7feb54169da - <rustc_metadata[980890dc6482dc21]::creader::CStore as rustc_session[9c2fdf61d4d3706c]::cstore::CrateStore>::def_path_hash_to_def_id
  13:      0x7feb5bf1bc6 - <rustc_middle[5be9dd9712efa76d]::ty::context::TyCtxt>::def_path_hash_to_def_id
  14:      0x7feb5c66cc4 - <rustc_query_system[4f119c27d31e4817]::dep_graph::dep_node::DepNode<rustc_middle[5be9dd9712efa76d]::dep_graph::dep_node::DepKind> as rustc_middle[5be9dd9712efa76d]::dep_graph::dep_node::DepNodeExt>::extract_def_id
  15:      0x7feb4e5d3a6 - <&[rustc_span[1644296338110723]::span_encoding::Span] as rustc_serialize[f2ace800d99ebe3b]::serialize::Decodable<rustc_query_impl[7231c5a387277fbd]::on_disk_cache::CacheDecoder>>::decode
  16:      0x7feb5bd60c4 - <rustc_middle[5be9dd9712efa76d]::ty::context::TyCtxt as rustc_query_system[4f119c27d31e4817]::dep_graph::DepContext>::try_load_from_on_disk_cache
  17:      0x7feb4f56dcd - <&[rustc_ast[84776166d75f4b05]::ast::InlineAsmTemplatePiece] as rustc_serialize[f2ace800d99ebe3b]::serialize::Decodable<rustc_query_impl[7231c5a387277fbd]::on_disk_cache::CacheDecoder>>::decode
  18:      0x7feb4ea2202 - <rustc_query_impl[7231c5a387277fbd]::on_disk_cache::OnDiskCache as rustc_middle[5be9dd9712efa76d]::ty::context::OnDiskCache>::drop_serialized_data
  19:      0x7feb5199b79 - rustc_incremental[70b4dd3adfb609db]::persist::work_product::delete_workproduct_files
  20:      0x7feb5179e9f - <rustc_incremental[70b4dd3adfb609db]::assert_dep_graph::GraphvizDepGraph as rustc_graphviz[d20dbb5caf23c65]::Labeller>::node_label
  21:      0x7feb5156758 - rustc_incremental[70b4dd3adfb609db]::persist::save::save_dep_graph
  22:      0x7feb1743f9a - <rustc_driver[d0f8e8bee9276d3d]::args::Error as core[caf1bcf4c9c99831]::fmt::Debug>::fmt
  23:      0x7feb170b960 - <rustc_codegen_ssa[8458bcb186c10200]::back::linker::BpfLinker as rustc_codegen_ssa[8458bcb186c10200]::back::linker::Linker>::no_default_libraries
  24:      0x7feb172ee76 - <rustc_codegen_ssa[8458bcb186c10200]::back::linker::BpfLinker as rustc_codegen_ssa[8458bcb186c10200]::back::linker::Linker>::no_default_libraries
  25:      0x7feb1737cd5 - <rustc_driver[d0f8e8bee9276d3d]::args::Error as core[caf1bcf4c9c99831]::fmt::Debug>::fmt
  26:      0x7feb172d42a - <rustc_codegen_ssa[8458bcb186c10200]::back::linker::BpfLinker as rustc_codegen_ssa[8458bcb186c10200]::back::linker::Linker>::no_default_libraries
  27:      0x7feb170e25a - <rustc_codegen_ssa[8458bcb186c10200]::back::linker::BpfLinker as rustc_codegen_ssa[8458bcb186c10200]::back::linker::Linker>::no_default_libraries
  28:      0x7feb17ae1e8 - rustc_driver[d0f8e8bee9276d3d]::pretty::print_after_hir_lowering
  29:      0x7fed774cbec - std::sys::windows::thread::Thread::new::h25c1e3623f62de88
  30:         0x76d3556d - BaseThreadInitThunk
  31:         0x76e9372d - RtlUserThreadStart

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.60.0-nightly (09cb29c64 2022-02-15) running on x86_64-pc-windows-msvc

note: compiler flags: --crate-type bin -C embed-bitcode=no -C codegen-units=1 -C debuginfo=2 -C incremental

@rustbot label +A-incr-comp +A-error-handling

[klensy]

@rustbot label +T-compiler +I-ICE

[Mark-Simulacrum]

I'm not sure that it makes sense to add checksums - they'll likely add overhead to compilation, and ultimately even source code could have bits flipped to be different in a way that rustc can't detect. I sort of see this as similar to writing random data into a running rustc's memory - it doesn't really feel in scope.

[klensy]

they'll likely add overhead to compilation

Yes, additional work, but how big?

writing random data into a running rustc's memory

Incremental cache is persistent between compilations, so it's not like playing with races in time of compilation.

I think i saw few ICEs when i stopped incremental build in the middle of work and restarted it (but there exist some sort of lock files, so not completed build shouldn't allow to restart it, yes?), so only way to fix it was cargo clean, but maybe it isn't related.

[Mark-Simulacrum]

cc #87896

Performance results from that attempt seem to suggest that the performance hit is pretty large for tiny crates (e.g., helloworld), but a not-insignificant 1-2% for larger crates -- without a clearer motivation (e.g., lots of spurious-seeming failures due to random bit flips), it seems like it's not something we'd probably want to take a hit on.

(I suspect the magnitude might be smaller if we used a faster hash function -- but probably not by that much).