Alternative _virtual_includes organization to solve MSVC issues #18683
Description
Description of the feature request:
Currently bazel constructs _virtual_includes paths for libraries in order to place files at known locations or restrict the set of files available by the compiler. These paths look somewhat like this, in the case of protobuf:
<bazel_output_base>/<hash>/execroot/<root workspace>/bazel-out/<config>/bin/external/com_google_protobuf/src/google/protobuf/compiler/objectivec/_virtual_includes/objectivec/google/protobuf/compiler/objectivec/text_format_decode_data.h
These constructed include paths are quite long, as they incorporate both the path to the bazel rule and the path to the file. Unfortunately MSVC uses a MAX_PATH of 260 characters, which leads to hard to discover errors such as:
In order to solve this, I request a feature to make _virtual_includes paths shorter.
If the virtual includes path were constructed more like:
<bazel_output_base>/<hash>/_virtual_includes/<different hash>/objectivec/google/protobuf/compiler/objectivec/text_format_decode_data.h
Then this problem would be essentially eliminated. The exact path construction doesn't matter, as long as it can eliminate much of the lengths of deep bazel rules.
This may require a rule context feature which, in essence, constructs a virtual path which encodes the entire build / workspace / package config.
Of course, links could be maintained from the current virtual include to that hash.
What underlying problem are you trying to solve with this feature?
Windows, but more importantly MSVC cl.exe, limits filename lengths to MAX_PATH (260) characters: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation https://developercommunity.visualstudio.com/t/clexe-compiler-driver-cannot-handle-long-file-path/975889
The bazel workspace and _virtual_includes repository construction runs into this limit. This has been a recurring issue in some windows builds as there are multiple ways to trigger it:
- C++ object files path is too long for some projects #15961
- Resolve Windows long path issue during C++ compilation #4149
- src build on windows not working protocolbuffers/protobuf#12947
The organization of the constructed virtual includes directories exacerbates this limit.
fatal error C1083: Cannot open include file: 'google/protobuf/compiler/objectivec/text_format_decode_data.h': No such file or directory
Workarounds involve setting the output_base startup flag, such as --output_base=C:\Out, which is not a universal solution, or reorganizing the directory structure within a repository, which cannot always be done on mature projects.
Which operating system are you running Bazel on?
Windows + MSVC (any version)
Any other information, logs, or outputs that you want to share?
For my specific build, which includes protobuf, this is an output of the @params file as supporting evidence. Here many of the bazel constructed paths are >130 characters, leaving 130 total for both the --bazel_root and the actual path.
If the bazel root is 'C:/users/laramiel/_bazel_laramiel/12345678', (which is 42 characters), that leaves at most 80 characters for the file + path in a bazel repository.
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/compiler/objectivec/_virtual_includes/objectivec
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/compiler/objectivec/_virtual_includes/line_consumer
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_nowkt
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_align
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/lite
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/port_def
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_allocation_policy
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_config
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_cleanup
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/string_block
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/varint_shuffle
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io_win32
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/gzip_stream
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/stubs
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/printer
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/zero_copy_sink
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/tokenizer
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/compiler/_virtual_includes/code_generator
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/compiler/objectivec/_virtual_includes/names
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/compiler/objectivec/_virtual_includes/names_internal
/Ibazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/descriptor_legacy