Thin LTO causes `relocation R_AARCH64_CALL26 out of range`

[emojifreak]

This could be a duplicate of #1394. When the kernel is too big, relocation out of range looks natural. On the other hand, CONFIG_RELOCATABLE=n should prevent such errors.

Besides, allyesconfig for arm64 leaves many modules disabled. This is because CONFIG_ARCH_*, CONFIG_*_MBOX and CONFIG_*_FIRMWARE are left disabled. Building a too big kernel needs manual configuration.

With the attached .config or reproducing script, linux 5.16-rc5 with CONFIG_RELOCATABLE=n causes the following build error:

  LD      .tmp_vmlinux.kallsyms1
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/ieee802154/adf7242.c:1263:(.text.adf7242_probe.0874045d102581664d600c7fc975d828+0x600): relocation R_AARCH64_CALL26 out of range: -134240376 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/refcount.h:227:(.text.__xfrm_state_lookup+0x1A4): relocation R_AARCH64_CALL26 out of range: -134253716 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/rcupdate.h:688:(.text.xfrm_replay_timer_handler.b959a7adba8b272d70e4565e7224e33c+0xB0): relocation R_AARCH64_CALL26 out of range: -134218960 is not in [-134217728, 134217727]
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/ieee802154/adf7242.c:1280:(.text.adf7242_probe.0874045d102581664d600c7fc975d828+0x954): relocation R_AARCH64_CALL26 out of range: -134241228 is not in [-134217728, 134217727]
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/ieee802154/adf7242.c:1250:(.text.adf7242_probe.0874045d102581664d600c7fc975d828+0x154C): relocation R_AARCH64_CALL26 out of range: -134244260 is not in [-134217728, 134217727]
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/ieee802154/adf7242.c:1167:(.text.adf7242_debugfs_init+0x68): relocation R_AARCH64_CALL26 out of range: -134247904 is not in [-134217728, 134217727]
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/ieee802154/ca8210.c:770:(.text.ca8210_spi_transfer_complete.7aa2a2ad9f21168c06e534838d7c2b4a+0x780): relocation R_AARCH64_CALL26 out of range: -134289224 is not in [-134217728, 134217727]
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/ieee802154/ca8210.c:770:(.text.ca8210_spi_transfer_complete.7aa2a2ad9f21168c06e534838d7c2b4a+0xDE0): relocation R_AARCH64_CALL26 out of range: -134290856 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/uaccess.h:200:(.text.ca8210_test_int_user_read.7aa2a2ad9f21168c06e534838d7c2b4a+0x35C): relocation R_AARCH64_CALL26 out of range: -134297908 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/uaccess.h:192:(.text.ca8210_test_int_user_write.7aa2a2ad9f21168c06e534838d7c2b4a+0x114): relocation R_AARCH64_CALL26 out of range: -134299548 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/rculist.h:157:(.text.hwsim_del_edge_nl.b824ffb7c1ef339108359d6dfbdb103e+0x420): relocation R_AARCH64_CALL26 out of range: -134344664 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/rculist.h:157:(.text.hwsim_add_one+0x868): relocation R_AARCH64_CALL26 out of range: -134349088 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/list.h:148:(.text.hwsim_del+0x48): relocation R_AARCH64_CALL26 out of range: -134357184 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/rculist.h:157:(.text.hwsim_del+0x110): relocation R_AARCH64_CALL26 out of range: -134357384 is not in [-134217728, 134217727]
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/wwan/wwan_core.c:332:(.text.__wwan_port_dev_assign_name+0x10C): relocation R_AARCH64_CALL26 out of range: -134362708 is not in [-134217728, 134217727]
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/wwan/wwan_core.c:960:(.text.wwan_register_ops+0x28C): relocation R_AARCH64_CALL26 out of range: -134366036 is not in [-134217728, 134217727]
ld.lld: error: /var/tmp/tmp28/arm64/linux-5.16-rc5/drivers/net/wwan/wwan_core.c:131:(.text.wwan_dev_name_match.dbd604d2a4f5b0e6b0fe8fe0e9224f26+0x60): relocation R_AARCH64_CALL26 out of range: -134373736 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/uaccess.h:200:(.text.wwan_port_fops_read.dbd604d2a4f5b0e6b0fe8fe0e9224f26+0x274): relocation R_AARCH64_CALL26 out of range: -134374492 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/uaccess.h:192:(.text.wwan_port_fops_write.dbd604d2a4f5b0e6b0fe8fe0e9224f26+0xD4): relocation R_AARCH64_CALL26 out of range: -134375084 is not in [-134217728, 134217727]
ld.lld: error: ./include/linux/uaccess.h:192:(.text.wwan_port_fops_at_ioctl+0xAC): relocation R_AARCH64_CALL26 out of range: -134381892 is not in [-134217728, 134217727]
ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
make[1]: *** [Makefile:1161: vmlinux] Error 1
make: *** [Makefile:1140: autoksyms_recursive] Error 2
make: Target 'all' not remade because of errors.

clang-build-arch28.sh.gz
config.txt.gz

[emojifreak]

By the way, I was unable to test CONFIG_LTO_CLANG_FULL with the above config because of shortage of the memory of my laptop.

[nathanchance]

Building a too big kernel needs manual configuration.

"Doctor, it hurts when I do this."

"So don't do that."

:)

But in all seriousness, the likelihood of this happening in the real world is fairly slim, as distributions will build modular kernels so that they can run on a wide variety of hardware and people who want a monolithic kernel are going to only build what they need for a single piece of hardware.

Does this require LTO to reproduce? I tried to build an arm64 allyesconfig + CONFIG_RELOCATABLE=n + CONFIG_LTO_CLANG_THIN=y kernel earlier and it was still linking two and a half hours in...

[emojifreak]

Does this require LTO to reproduce?

@nathanchance Yes, it does. The symptom disappears with LTO_NONE=y. This seems an LTO problem. Probably because of this, I cannot produce a similar symptom with 32-bit ARM even if all modules are built-in.

I completely agree that allyesconfig is purely test purpose. It seems useful here for uncovering an LTO problem...

[nickdesaulniers]

LLD accepts --thinlto-jobs=<value> and --threads=<value> to limit the number of concurrent jobs (and thus resident set size) when linking.

[emojifreak]

No-LTO does not show this symptom even with CONFIG_RELOCATABLE=y. I changed the issue title.

[emojifreak]

shadow call stack, CFI, DEBUG_INFO, TRIM_UNUSED_KSYMS are unnecessary to cause this symptom. The script attached to #1554 still causes this.

[emojifreak]

This might be caused by significant differences in sizes of vmlinux:

Without LTO:

$ LANG=C ls -l *vmlin* .*vmlinu*
-rwxr-xr-x 1 ryutaroh ryutaroh  892729552 Dec 26 08:56 .tmp_vmlinux.kallsyms1
-rw-r--r-- 1 ryutaroh ryutaroh  126627909 Dec 26 08:57 .tmp_vmlinux.kallsyms1.S
-rw-r--r-- 1 ryutaroh ryutaroh  219577230 Dec 26 08:56 .tmp_vmlinux.kallsyms1.map
-rw-r--r-- 1 ryutaroh ryutaroh   21151720 Dec 26 08:57 .tmp_vmlinux.kallsyms1.o
-rwxr-xr-x 1 ryutaroh ryutaroh  913922360 Dec 26 08:57 .tmp_vmlinux.kallsyms2
-rw-r--r-- 1 ryutaroh ryutaroh  126673471 Dec 26 08:57 .tmp_vmlinux.kallsyms2.S
-rw-r--r-- 1 ryutaroh ryutaroh  219578151 Dec 26 08:57 .tmp_vmlinux.kallsyms2.map
-rw-r--r-- 1 ryutaroh ryutaroh   21159320 Dec 26 08:57 .tmp_vmlinux.kallsyms2.o
-rwxr-xr-x 1 ryutaroh ryutaroh  913922360 Dec 26 08:58 .tmp_vmlinux.kallsyms3
-rw-r--r-- 1 ryutaroh ryutaroh  126673471 Dec 26 08:58 .tmp_vmlinux.kallsyms3.S
-rw-r--r-- 1 ryutaroh ryutaroh  219578151 Dec 26 08:58 .tmp_vmlinux.kallsyms3.map
-rw-r--r-- 1 ryutaroh ryutaroh   21159320 Dec 26 08:58 .tmp_vmlinux.kallsyms3.o
-rw-r--r-- 1 ryutaroh ryutaroh       1179 Dec 26 08:59 .vmlinux.cmd
-rwxr-xr-x 1 ryutaroh ryutaroh  914061376 Dec 26 08:58 vmlinux
-rw-r--r-- 1 ryutaroh ryutaroh  219579734 Dec 26 08:58 vmlinux.map
-rw-r--r-- 1 ryutaroh ryutaroh 2510581888 Dec 26 08:56 vmlinux.o
-rw-r--r-- 1 ryutaroh ryutaroh    1543609 Dec 26 08:56 vmlinux.symvers

With thin LTO:

$ LANG=C ls -l *vmlin* .*vmlinu*
-rw-r--r-- 1 ryutaroh ryutaroh 1814001680 Dec 26 16:13 .tmp_vmlinux.kallsyms1.map
-rw-r--r-- 1 ryutaroh ryutaroh 5077944904 Dec 26 16:11 vmlinux.o
-rw-r--r-- 1 ryutaroh ryutaroh    1543609 Dec 26 16:11 vmlinux.symvers

The difference ratio is even greater than the x86 case #1556.