MSVC Linker Issue When Targeting ARM64EC

[JiachenYuan]

Hi! A recent commit has caused a linker issue when cross-compiling the LLVM project targeting ARM64EC on a AMD64 host Windows machine. The reduced steps to reproduce the linker error is listed below.

Prerequisite

MSVC

• Visual Studio Community 2022 17.14.4

Individual MSVC components:

• MSVC v143 - VS 2022 C++ ARM64/ARM64EC build tools (Latest)
• C++ ATL for latest v143 build tools (ARM64/ARM64EC)
• C++ ATL for latest v143 build tools (x86 & x64)

Build for arm64ec

1. Open a x64 Native Tools Command Prompt for VS 2022
2. The linker error can be reproduced by

git clone https://github.com/llvm/llvm-project.git
mkdir build
cd build

cmake -G "Visual Studio 17 2022" -A ARM64EC -DLLVM_ENABLE_PROJECTS="llvm" -DLLVM_TARGETS_TO_BUILD="AArch64" -DLLVM_OPTIMIZED_TABLEGEN=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -DCMAKE_CXX_FLAGS="/wd4146 /wd4244 /wd4267 /wd4996 /wd4800 /D_HAS_EXCEPTIONS=0" -DCMAKE_EXE_LINKER_FLAGS="softintrin.lib arm64rt.lib" -DCMAKE_SHARED_LINKER_FLAGS="softintrin.lib arm64rt.lib" -DCMAKE_MODULE_LINKER_FLAGS="softintrin.lib arm64rt.lib" -DLLVM_DISABLE_ASSEMBLY_FILES=ON  ..\llvm-project\llvm

msbuild LLVM.sln /m /p:Platform=ARM64EC /p:Configuration=Release

(The -DLLVM_DISABLE_ASSEMBLY_FILES flag is suggested in #100335 to ignore problem: LNK1372: hybrid map entry target function 'llvm_blake3_compress_in_place_sse41' is not 4-byte aligned)

Linker Error Message

LINK : error LNK2001: unresolved external symbol "??_C@_0L@OHILDFJG@" (??_C@_0L@OHILDFJG@) (EC Symbol) 

Triage

Dumping the symbol tables of the generated object files shows that the unresolved symbol is probably related to the HybridPatchableTargetSuffix global variable in AsmPrinter.cpp introduced by the aforementioned commit. Directly substituting the literal strings to its uses does not solve the problem. A dirty workaround that works for me is to divide up the string at its definition and re-concatenate the parts at the original uses.

[efriedma-quic]

CC @dpaoliello

When you say "A recent commit has caused a linker issue", did you run a bisect?

If you're building using MSVC compiler and linker, a weird link error could be an MSVC compiler/linker bug. I don't recognize that symbol name, and undname doesn't think it's a valid symbol mangling.

Is there any other log output besides the error message? Normally I think the linker dumps some info about where the symbol was defined. What's the link command?

[JiachenYuan]

Thank you for helping!

I didn't run git bisect, and I also think it might be an MSVC bug as I have never seen something like this before. I am wondering if there this is a valid motivation to have a workaround of the issue.

The steps I picked out that particular commit is:

1. Using undname to see what symbol the linker tries to find. It gives back the string as is, which is not helpful.
2. Using dumpbin /symbols XXX | find "??_C@_0L@OHILDFJG@" over all generated obj files to see who is referring to this symbol. Found AsmPrinter` is looking for the symbol.
3. Since it started fairly recently, I looked at the commit history of AsmPrinter.cpp (at the time of issue discovery, there were only about three more new commits on top of the mentioned one), and I reverted each of them individually. After reverting the mentioned commit, the error disappeared.
4. Played around the variable definition as described in the Triage section above.

The error does not come with any more useful message, even on verbose mode. It was presented on its own. You can build it locally to see the weird behavior.

[efriedma-quic]

Looking a bit more, ??_C@_0L@OHILDFJG@ is a weird version of the string literal "$hp_target", I think. At least, the CRC is right... normally the symbol name isn't truncated like that. I guess MSVC doesn't like constexpr std::string_view HybridPatchableTargetSuffix = "$hp_target";? Maybe see if it helps to replace it with #define HybridPatchableTargetSuffix "$hp_target".

In any case, please file a bug against MSVC at https://developercommunity.visualstudio.com/cpp .

[jameswplayer]

While an MSVC bug is called for, I think a work-around in LLVM code is also appropriate given that this bug occurs in many versions of the MSVC linker.

[efriedma-quic]

Adding workarounds to LLVM for host compiler bugs is fine. But we should have a comment explicitly stating it's a workaround, with a reference to an upstream bug report.

[efriedma-quic]

Oh, I just realized what's probably happening: the compiler is producing a symbol something like ??_C@_0L@OHILDFJG@$hp_target@, this then triggers hybrid_patchable processing in the linker, which truncates the symbol name before the first dollar-sign, then it tries to find a symbol which doesn't exist.

CC @cjacek. (I think lld's implementation of hybrid_patchable doesn't have this issue, but FYI.)

[JiachenYuan]

Yeah, it might be the cause. I just remember seeing the string you pasted while using dumpbin. Anyway, I submitted a MSVC bug. Dropping the link here for reference: https://developercommunity.visualstudio.com/t/MSVC-Linker-Issue-When-Cross-Compiling-L/10920141.

I have a workaround in our downstream and will bring up a PR with a relevant comment.

[efriedma-quic]

Workaround merged in #143659 . I'll leave this open for the moment to track what Microsoft does.