LLD and `-flto=thin` calls `ldc.register_dso` multiple times instead of once at startup/shutdown

[JohanEngelen]

When using LLD with -flto=thin, we are running into the problem that each D module's call to ldc.register_dso in llvm.global_ctors is not folded into one call. We are relying on LLVM functionality where if a D module's ldc.register_dso is "discarded" because of another module's ldc.register_dso overriding it, then the entry in llvm.global_ctors is also discarded. This is apparently happening with ld.gold, but not with ld.lld.

It's related to this work: #2870

Strangely, when I manually put the ctor/dtor in .init_array/.fini_array sections, the symbols are not merged either (not with bfd, gold, or lld). That is, replacing

ldc/gen/modules.cpp

Lines 367 to 368 in 2ec2fd7

	llvm::appendToGlobalCtors(gIR->module, registerDSO, 65535, registerDSO);
	llvm::appendToGlobalDtors(gIR->module, registerDSO, 65535, registerDSO);

with

    const auto dsoCtor = defineDSOGlobal("ldc.dso_ctor", registerDSO);
    const auto dsoDtor = defineDSOGlobal("ldc.dso_dtor", registerDSO);
    gIR->usedArray.push_back(dsoCtor);
    gIR->usedArray.push_back(dsoDtor);
    dsoCtor->setSection(".init_array");
    dsoDtor->setSection(".fini_array");

[JohanEngelen]

(testcase = https://github.com/ldc-developers/ldc/blob/master/tests/linking/thinlto_modulecdtors.d)

[kinke]

Sounds more like an LLD issue to me. FWIW, I think the way to tackle and further simplify this would be to adapt the Windows scheme with a special prebuilt module: https://github.com/ldc-developers/druntime/blob/ldc/src/rt/dso_windows.d

GDC does a similar thing apparently. All code would be in druntime, and people wouldn't wonder about the extra emissions when inspecting produced asm either. All we'd lose is the ability to link arbitrary D object files to binaries when bypassing LDC for linking, in which case the extra module would need to be linked in manually.

[JohanEngelen]

Reproducer for LLVM (will submit to LLVM and ask about it):

; File: main.ll
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

@str.ctor = private unnamed_addr constant [6 x i8] c"ctor\0A\00"
@llvm.global_ctors = appending global [1 x { i32, void ()*, i8* }] [{ i32, void ()*, i8* } { i32 65535, void ()* @module_ctor, i8* bitcast (void ()* @module_ctor to i8*) }]

declare i32 @printf(i8*, ...)

define linkonce_odr hidden void @module_ctor() {
  %1 = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([6 x i8], [6 x i8]* @str.ctor, i64 0, i64 0))
  ret void
}

define i32 @main(i32 %argc_arg, i8** %argv_arg) {
  ret i32 0
}

; File: second_object.ll
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

@str.ctor = private unnamed_addr constant [6 x i8] c"ctor\0A\00"
@llvm.global_ctors = appending global [1 x { i32, void ()*, i8* }] [{ i32, void ()*, i8* } { i32 65535, void ()* @module_ctor, i8* bitcast (void ()* @module_ctor to i8*) }]

declare i32 @printf(i8*, ...)

define linkonce_odr hidden void @module_ctor() {
  %1 = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([6 x i8], [6 x i8]* @str.ctor, i64 0, i64 0))
  ret void
}

Script file to build and reproduce:

# File: script.sh
LLC=/home/johan/llvm/install12/bin/llc
OPT=/home/johan/llvm/install12/bin/opt
LLVMgold=/home/johan/llvm/install12/lib/LLVMgold.so

$LLC second_object.ll -O3 --relocation-model=pic -filetype=obj -o second_object.o
$LLC main.ll -O3 --relocation-model=pic -filetype=obj -o main.o
cc main.o second_object.o -fuse-ld=gold -o bug_gold
cc main.o second_object.o -fuse-ld=lld -o bug_lld

$OPT main.ll -O3 -o main_thinlto.o
cc main_thinlto.o second_object.o -o bug_thinlto_gold -fuse-ld=gold -Wl,-plugin,$LLVMgold -Wl,-plugin-opt=thinlto -Wl,-plugin-opt=-function-sections -Wl,-plugin-opt=-data-sections

$OPT main.ll -O3 -o main_thinlto.o
cc main_thinlto.o second_object.o -o bug_thinlto_lld -fuse-ld=lld #-Wl,-plugin,$LLVMgold -Wl,-plugin-opt=thinlto -Wl,-plugin-opt=-function-sections -Wl,-plugin-opt=-data-sections

echo "bug_gold:"
./bug_gold
echo "bug_lld:"
./bug_lld
echo "bug_thinlto_gold:"
./bug_thinlto_gold
echo "bug_thinlto_lld:"
./bug_thinlto_lld

Output:

❯ ./script.sh
bug_gold:
ctor
bug_lld:
ctor
bug_thinlto_gold:
ctor
bug_thinlto_lld:
ctor
ctor

The double "ctor" at the end for bug_thinlto_lld is the bug.
Tested with LLVM 12.

[JohanEngelen]

https://bugs.llvm.org/show_bug.cgi?id=51067

[MaskRay]

I left comments on https://bugs.llvm.org/show_bug.cgi?id=51067

tl;dr The third field of @llvm.global_ctors only works for PE/COFF. Don't use it for other binary formats (ELF, Mach-O, wasm).
There is currently no way to make a SHT_INIT_ARRAY section GCable with LLD/GNU ld.
[...]

Actually, I think printing twice is the reasonable behavior, if the linker cannot GC .init_array input sections.
I'll not call "printing once" wrong because the IR symbol table does not report that module_ctor in main.bc is a comdat.
The IR doesn't have $module_ctor = comdat any, but the emitted assembly has comdat semantics.

Ultimately, the issue is that llvm/llvm-project@c4b4253 the ELF implementation in llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp does not do what LangRef documents.

[dnadlinger]

It's been a long while since I worked on this, but from memory, the module registry implementation assumes that the constructor can be called an arbitrary number of times and correctly does the lazy initialisation internally, doesn't it?

[dnadlinger]

Ah, right, it was "simplified" since – have fun dealing with this mess!

[JohanEngelen]

I think the way to tackle and further simplify this would be to adapt the Windows scheme with a special prebuilt module: https://github.com/ldc-developers/druntime/blob/ldc/src/rt/dso_windows.d

I agree.