LLVM commit a79995ca6004 breaks as-instr and as-option for ARCH=arm on older kernels

[nathanchance]

CI reports several build failures due to the smc instruction when building ARCH=arm multi_v7_defconfig with LLVM 17 on linux-5.4.y and earlier.

https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/5306568615/jobs/9604484659

$ make -skj"$(nproc)" ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- LLVM=1 LLVM_IAS=0 mrproper multi_v7_defconfig drivers/firmware/qcom_scm-32.o
/tmp/qcom_scm-32-c413c1.s: Assembler messages:
/tmp/qcom_scm-32-c413c1.s:56: Error: selected processor does not support `smc #0' in ARM mode
/tmp/qcom_scm-32-c413c1.s:70: Error: selected processor does not support `smc #0' in ARM mode
/tmp/qcom_scm-32-c413c1.s:178: Error: selected processor does not support `smc #0' in ARM mode
/tmp/qcom_scm-32-c413c1.s:408: Error: selected processor does not support `smc #0' in ARM mode
/tmp/qcom_scm-32-c413c1.s:562: Error: selected processor does not support `smc #0' in ARM mode
/tmp/qcom_scm-32-c413c1.s:947: Error: selected processor does not support `smc #0' in ARM mode
/tmp/qcom_scm-32-c413c1.s:1087: Error: selected processor does not support `smc #0' in ARM mode
/tmp/qcom_scm-32-c413c1.s:1134: Error: selected processor does not support `smc #0' in ARM mode
clang: error: assembler command failed with exit code 1 (use -v to see invocation)

I bisected this to llvm/llvm-project@a79995c:

# bad: [8e1f820bb4eadf5c0704818f6063e0db1006e32d] [dsymutil] Update DWARF 5 tests after 8119ab9bf78b
# good: [168fa3181614fc7bc33e0b8c588e3ec74b3f65b5] [libc] Fix some tests on NVPTX due to insufficient stack size
git bisect start '8e1f820bb4eadf5c0704818f6063e0db1006e32d' '168fa3181614fc7bc33e0b8c588e3ec74b3f65b5'
# good: [a48fe898857c95a063fa6c201343dca969bc098a] [mlir][ArmSME] Add initial dialect with basic lowering of vector.transfer write to zero
git bisect good a48fe898857c95a063fa6c201343dca969bc098a
# good: [aee3a9f5bcdc13de4bb1b3eb04845b172fffb23c] [libc++][NFC] clang-format new_handler.cpp
git bisect good aee3a9f5bcdc13de4bb1b3eb04845b172fffb23c
# good: [11ebe3d906558d93a607347de472e7718127f409] [RISCV] relaxDwarfCallFrameFragment: remove unneeded relocations for relaxation
git bisect good 11ebe3d906558d93a607347de472e7718127f409
# good: [0a246a0c723dc94b77e83279d76c30010cdc2fc0] [LV] Use VPValues when creating GEP with all invariant indices.
git bisect good 0a246a0c723dc94b77e83279d76c30010cdc2fc0
# bad: [0356ceedf2e90459597adfda19db2c52faf7aacd] [DebugInfo] Change DWARFDebugAbbrev initialization
git bisect bad 0356ceedf2e90459597adfda19db2c52faf7aacd
# bad: [8a403166aa617d66bf420ec11350e9e0f4d1f02f] [RISCV] Treat __riscv_vsetvl_*(-1) as vlmax.
git bisect bad 8a403166aa617d66bf420ec11350e9e0f4d1f02f
# bad: [61c9052cec10a80f9a48d2da9e443629089c9a88] [libc] Add LIBC_INLINE_VAR for inline variables
git bisect bad 61c9052cec10a80f9a48d2da9e443629089c9a88
# bad: [c8f4ba374b10b9851a580ad73de8bd4d8e40be60] [NFC] Autogenerate CodeGen/ARM/vlddup.ll
git bisect bad c8f4ba374b10b9851a580ad73de8bd4d8e40be60
# bad: [a79995ca6004082774a87f7a58ab6be5343364b7] [Driver] Allow warning for unclaimed TargetSpecific options
git bisect bad a79995ca6004082774a87f7a58ab6be5343364b7
# good: [ec77747fbdca901e0fded58f940dae62e0f6b726] [NFC] Autogenerate CodeGen/ARM/sub-cmp-peephole.ll
git bisect good ec77747fbdca901e0fded58f940dae62e0f6b726
# first bad commit: [a79995ca6004082774a87f7a58ab6be5343364b7] [Driver] Allow warning for unclaimed TargetSpecific options

Comparing the command of drivers/firmware/qcom_scm-32.o before and after the change reveals -DREQUIRES_SEC is getting dropped:

diff --git a/tmp/.psub.q4ut5Iov1j b/tmp/.psub.NTsX2cJIis
index c9fa288ba7ef..ae3237ee51f9 100644
--- a/tmp/.psub.q4ut5Iov1j
+++ b/tmp/.psub.NTsX2cJIis
@@ -1 +1 @@
-  ccache /mnt/nvme/tmp/cvise.KRtce4djrq/install/llvm-good/bin/clang -Wp,-MD,drivers/firmware/.qcom_scm-32.o.d -nostdinc -isystem /mnt/nvme/tmp/cvise.KRtce4djrq/install/llvm-good/lib/clang/17/include -I./arch/arm/include -I./arch/arm/include/generated  -I./include -I./arch/arm/include/uapi -I./arch/arm/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -mlittle-endian -Qunused-arguments -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu89 --target=arm-linux-gnueabi --prefix=/home/nathan/cbl/toolchains/binutils-latest/bin/arm-linux-gnueabi- --gcc-toolchain=/home/nathan/cbl/toolchains/binutils/2023-06-20_22-22-26-00a854287778c852191df9f9815c5d601daa5ae1 -no-integrated-as -Werror=unknown-warning-option -fno-dwarf2-cfi-asm -mabi=aapcs-linux -mfpu=vfp -funwind-tables -meabi gnu -marm -Wa,-mno-warn-deprecated -D__LINUX_ARM_ARCH__=7 -march=armv7-a -msoft-float -Uarm -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=1024 -fstack-protector-strong -Wno-format-invalid-specifier -Wno-gnu -Wno-tautological-compare -mno-global-merge -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-array-bounds -fno-strict-overflow -fno-merge-all-constants -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -fmacro-prefix-map=./= -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-unaligned-access -Wno-cast-function-type-strict -DREQUIRES_SEC=1 -march=armv7-a    -DKBUILD_BASENAME='"qcom_scm_32"' -DKBUILD_MODNAME='"qcom_scm_32"' -c -o drivers/firmware/qcom_scm-32.o drivers/firmware/qcom_scm-32.c
+  ccache /mnt/nvme/tmp/cvise.KRtce4djrq/install/llvm-bad/bin/clang -Wp,-MD,drivers/firmware/.qcom_scm-32.o.d -nostdinc -isystem /mnt/nvme/tmp/cvise.KRtce4djrq/install/llvm-bad/lib/clang/17/include -I./arch/arm/include -I./arch/arm/include/generated  -I./include -I./arch/arm/include/uapi -I./arch/arm/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -mlittle-endian -Qunused-arguments -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu89 --target=arm-linux-gnueabi --prefix=/home/nathan/cbl/toolchains/binutils-latest/bin/arm-linux-gnueabi- --gcc-toolchain=/home/nathan/cbl/toolchains/binutils/2023-06-20_22-22-26-00a854287778c852191df9f9815c5d601daa5ae1 -no-integrated-as -Werror=unknown-warning-option -fno-dwarf2-cfi-asm -mabi=aapcs-linux -mfpu=vfp -funwind-tables -meabi gnu -marm -Wa,-mno-warn-deprecated -D__LINUX_ARM_ARCH__=7 -march=armv7-a -msoft-float -Uarm -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=1024 -fstack-protector-strong -Wno-format-invalid-specifier -Wno-gnu -Wno-tautological-compare -mno-global-merge -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-array-bounds -fno-strict-overflow -fno-merge-all-constants -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -fmacro-prefix-map=./= -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-unaligned-access -Wno-cast-function-type-strict -march=armv7-a    -DKBUILD_BASENAME='"qcom_scm_32"' -DKBUILD_MODNAME='"qcom_scm_32"' -c -o drivers/firmware/qcom_scm-32.o drivers/firmware/qcom_scm-32.c

Dumping the as-instr invocation from drivers/firmware/Makefile reveals:

clang: error: unsupported option '-mabi=' for target 'arm-linux-gnueabi'

-mabi=aapcs-linux is added to both KBUILD_CFLAGS and KBUILD_AFLAGS but it errors with -x assembler, which as-instr and as-option use on older kernels, whereas the actual .S build step will use -x assembler-with-cpp internally.

$ clang --target=arm-linux-gnueabi -mabi=aapcs-linux -x c -c -o /dev/null /dev/null

$ clang --target=arm-linux-gnueabi -mabi=aapcs-linux -x assembler -c -o /dev/null /dev/null
clang: error: unsupported option '-mabi=' for target 'arm-linux-gnueabi'

$ clang --target=arm-linux-gnueabi -mabi=aapcs-linux -x assembler-with-cpp -c -o /dev/null /dev/null

The error above is not visible with 5.10 and newer due to commit 0224898152bd ("firmware: qcom_scm-32: Use SMC arch wrappers"), which may be backportable as a solution to that one error. As mentioned above, there are other errors on older branches for similar reasons, so it may be better to fix this in a more universal way.

We may be able to move older releases to using -x assembler-with-cpp like we have with newer versions:

diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 5d247d8f1e04..9b37744addda 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -108,7 +108,7 @@ as-option = $(call try-run,\
 # Usage: cflags-y += $(call as-instr,instr,option1,option2)

 as-instr = $(call try-run,\
-       printf "%b\n" "$(1)" | $(CC) $(KBUILD_AFLAGS) -c -x assembler -o "$$TMP" -,$(2),$(3))
+       printf "%b\n" "$(1)" | $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_AFLAGS) -c -x assembler-with-cpp -o "$$TMP" -,$(2),$(3))

 # __cc-option
 # Usage: MY_CFLAGS += $(call __cc-option,$(CC),$(MY_CFLAGS),-march=winchip-c6,-march=i586)

But I see another internal error on 5.4, there must have been some additional Kbuild refactoring done at some point to avoid this on newer releases:

<built-in>:1:10: fatal error: 'asm/unified.h' file not found
    1 | #include "asm/unified.h"
      |          ^~~~~~~~~~~~~~~
1 error generated.

which comes from -include asm/unified.h in arch/arm/Makefile's KBUILD_AFLAGS. I will continue to look at ways to workaround this on the Linux side but it would be nice if -mabi= would not error with -x assembler in the first place, although I understand the error is pointing out that the flag does nothing for that particular target (even though GCC won't warn).

cc @MaskRay

[MaskRay]

The -mabi= option for assembler seems strange. GCC translates -mabi={apcs-gnu,atpcs} to gas -meabi=gnu and other -mabi= values to -meabi=5. To the best of my knowledge, LLVM integrated assembler don't support setting e_flags to any value other than EF_ARM_EABI_VER5.

LLVM integrated assembler does not handle -mabi= and there was a warning before.
https://reviews.llvm.org/D153691 will change back the error to a warning.
It's still recommended to drop -mabi= option for -x assembler.

[nathanchance]

Thanks, llvm/llvm-project@fe5bab5 helps avoid this issue when using the integrated assembler but it does not appear to resolve the issue when using -fno-integrated-as.

$ clang --version | head -1
ClangBuiltLinux clang version 17.0.0 (https://github.com/llvm/llvm-project cb1911a2db14b02a3f7855723d9138d26c41e66f)

$ clang --target=arm-linux-gnueabi -mabi=aapcs-linux -x assembler -c -o /dev/null /dev/null
clang: warning: argument unused during compilation: '-mabi=aapcs-linux' [-Wunused-command-line-argument]

$ clang --target=arm-linux-gnueabi -fno-integrated-as -mabi=aapcs-linux -x assembler -c -o /dev/null /dev/null
clang: error: unsupported option '-mabi=' for target 'arm-linux-gnueabi'

[MaskRay]

Sorry for missing the notification. I don't have a good personal filtering system for https://github.com/notifications and it is full of items (mostly llvm-project issues), so I easily miss notifications that I should pay attention to...

$ clang --target=arm-linux-gnueabi -fno-integrated-as -mabi=aapcs-linux -x assembler -c -o /dev/null /dev/null
clang: error: unsupported option '-mabi=' for target 'arm-linux-gnueabi'

Sent https://reviews.llvm.org/D154736 to downgrade the error to clang: warning: argument unused during compilation: '-mabi=aapcs-linux' [-Wunused-command-line-argument]