Reactant_jll v0.0.237

[avik-pal]

No description provided.

[avik-pal]

https://buildkite.com/julialang/yggdrasil/builds/23051/steps/canvas?sid=0198f799-dc7a-49ce-ac5e-b1cb13d3e147#0198f799-dccc-44c4-8d21-2c44672897a7/1098-6580

[giordano]

https://buildkite.com/julialang/yggdrasil/builds/23051/steps/canvas?sid=0198f799-dc7a-49ce-ac5e-b1cb13d3e147#0198f799-dccc-44c4-8d21-2c44672897a7/1098-6580

For future reference, since the link is ephemeral, and the content non-searchable from here, the error is

[23:41:46] ERROR: /workspace/bazel_root/097636303b1142f44508c1d8e3494e4b/external/xla/xla/stream_executor/cuda/BUILD:2150:13: Compiling xla/stream_executor/cuda/all_reduce_kernel_cuda.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing CppCompile command (from target @@xla//xla/stream_executor/cuda:all_reduce_kernel_cuda) 
[23:41:46]   (cd /workspace/bazel_root/097636303b1142f44508c1d8e3494e4b/sandbox/processwrapper-sandbox/9069/execroot/__main__ && \
[23:41:46]   exec env - \
[23:41:46]     CLANG_CUDA_COMPILER_PATH=/opt/bin/x86_64-linux-gnu-libgfortran5-cxx11-cuda_version+12.4-gpu+cuda-mode+opt/clang \
[23:41:46]     GRPC_BAZEL_RUNTIME=1 \
[23:41:46]     LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/lib/csl-glibc-x86_64:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib:/workspace/destdir/lib64:/workspace/destdir/lib \
[23:41:46]     PATH=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.local/bin:/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.local/bin:/opt/bin/x86_64-linux-gnu-libgfortran5-cxx11-cuda_version+12.4-gpu+cuda-mode+opt:/opt/x86_64-linux-gnu/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin \
[23:41:46]     PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb \
[23:41:46]     PWD=/proc/self/cwd \
[23:41:46]     TEMP=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[23:41:46]     TMP=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[23:41:46]     TMPDIR=/workspace/srcdir/Reactant.jl/deps/ReactantExtra/.tmp \
[23:41:46]   external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/external/xla/xla/stream_executor/cuda/_objs/all_reduce_kernel_cuda/all_reduce_kernel_cuda.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/xla/xla/stream_executor/cuda/_objs/all_reduce_kernel_cuda/all_reduce_kernel_cuda.pic.o' '-DGOOGLE_PROTOBUF_USING_BAZEL=1' '-DEIGEN_MAX_ALIGN_BYTES=64' -DEIGEN_ALLOW_UNALIGNED_SCALARS '-DEIGEN_USE_AVX512_GEMM_KERNELS=0' -DHAVE_SYS_UIO_H -DTF_USE_SNAPPY -DTENSORFLOW_USE_NUMA '-DLLVM_ON_UNIX=1' '-DNO_HAVE_BACKTRACE=0' '-DBACKTRACE_HEADER=<execinfo.h>' '-DLTDL_SHLIB_EXT=".so"' '-DLLVM_PLUGIN_EXT=".so"' '-DLLVM_ENABLE_LLVM_EXPORT_ANNOTATIONS=1' '-DLLVM_ENABLE_PLUGINS=1' '-DLLVM_ENABLE_THREADS=0' '-DHAVE_DEREGISTER_FRAME=1' '-DHAVE_LIBPTHREAD=1' '-DFAKE_HAVE_PTHREAD_GETNAME_NP=0' '-DHAVE_PTHREAD_H=1' '-DFAKE_HAVE_PTHREAD_SETNAME_NP=0' '-DHAVE_REGISTER_FRAME=1' '-DHAVE_SETENV_R=1' '-DHAVE_STRERROR_R=1' '-DHAVE_SYSEXITS_H=1' '-DHAVE_SYS_IOCTL_H=1' '-DHAVE_UNISTD_H=1' -D_GNU_SOURCE '-DHAVE_GETAUXVAL=1' '-DDONT_HAVE_ANY_MALLINFO=0' '-DHAVE_SBRK=1' '-DHAVE_STRUCT_STAT_ST_MTIM_TV_NSEC=1' -DHAVE_BUILTIN_THREAD_POINTER '-DLLVM_NATIVE_ARCH="X86"' '-DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser' '-DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter' '-DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler' '-DLLVM_NATIVE_TARGET=LLVMInitializeX86Target' '-DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo' '-DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC' '-DLLVM_NATIVE_TARGETMCA=LLVMInitializeX86TargetMCA' '-DLLVM_HOST_TRIPLE="x86_64-unknown-linux-gnu"' '-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-linux-gnu"' '-DLLVM_VERSION_MAJOR=22' '-DLLVM_VERSION_MINOR=0' '-DLLVM_VERSION_PATCH=0' '-DLLVM_VERSION_STRING="22.0.0git"' -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS '-DLLVM_HAS_AMDGPU_TARGET=1' '-DLLVM_HAS_NVPTX_TARGET=1' '-DLLVM_HAS_AArch64_TARGET=1' '-DLLVM_HAS_X86_TARGET=1' '-DLLVM_HAS_ARM_TARGET=1' '-DBLAKE3_USE_NEON=0' -DBLAKE3_NO_AVX2 -DBLAKE3_NO_AVX512 -DBLAKE3_NO_SSE2 -DBLAKE3_NO_SSE41 '-DCLANG_CUDA=1' -iquote external/xla -iquote bazel-out/k8-opt/bin/external/xla -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/tsl -iquote bazel-out/k8-opt/bin/external/tsl -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/ml_dtypes_py -iquote bazel-out/k8-opt/bin/external/ml_dtypes_py -iquote external/snappy -iquote bazel-out/k8-opt/bin/external/snappy -iquote external/hwloc -iquote bazel-out/k8-opt/bin/external/hwloc -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/highwayhash -iquote bazel-out/k8-opt/bin/external/highwayhash -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -iquote external/llvm-project -iquote bazel-out/k8-opt/bin/external/llvm-project -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/cuda_cudart -iquote bazel-out/k8-opt/bin/external/cuda_cudart -iquote external/cuda_cublas -iquote bazel-out/k8-opt/bin/external/cuda_cublas -iquote external/cuda_cccl -iquote bazel-out/k8-opt/bin/external/cuda_cccl -iquote external/cuda_nvtx -iquote bazel-out/k8-opt/bin/external/cuda_nvtx -iquote external/cuda_nvcc -iquote bazel-out/k8-opt/bin/external/cuda_nvcc -iquote external/cuda_cusolver -iquote bazel-out/k8-opt/bin/external/cuda_cusolver -iquote external/cuda_cufft -iquote bazel-out/k8-opt/bin/external/cuda_cufft -iquote external/cuda_cusparse -iquote bazel-out/k8-opt/bin/external/cuda_cusparse -iquote external/cuda_curand -iquote bazel-out/k8-opt/bin/external/cuda_curand -iquote external/cuda_cupti -iquote bazel-out/k8-opt/bin/external/cuda_cupti -iquote external/cuda_nvml -iquote bazel-out/k8-opt/bin/external/cuda_nvml -iquote external/cuda_nvjitlink -iquote bazel-out/k8-opt/bin/external/cuda_nvjitlink -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/internal_visibility -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/micro_string -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_align -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/port -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/lite -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_allocation_policy -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_cleanup -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/string_block -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/endian -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/varint_shuffle -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io_win32 -Ibazel-out/k8-opt/bin/external/com_google_protobuf/third_party/utf8_range/_virtual_includes/utf8_validity -Ibazel-out/k8-opt/bin/external/com_google_protobuf/third_party/utf8_range/_virtual_includes/utf8_range -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/gzip_stream -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/stubs -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/printer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/zero_copy_sink -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/tokenizer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/any_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/api_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/source_context_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/type_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/compiler/_virtual_includes/plugin_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/descriptor_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/duration_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/empty_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/field_mask_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/struct_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/timestamp_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/wrappers_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_layering_check_legacy -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/compiler/_virtual_includes/importer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/delimited_message_util -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/differencer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/field_mask_util -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/json_util -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/json -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/parser -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/descriptor_traits -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/lexer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/message_path -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/zero_copy_buffered_stream -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/untyped_message -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/type_resolver -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/descriptor_legacy -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/unparser -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/writer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/time_util -Ibazel-out/k8-opt/bin/external/llvm-project/third-party/siphash/_virtual_includes/siphash -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers -Ibazel-out/k8-opt/bin/external/cuda_cudart/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cublas/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cccl/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvtx/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvcc/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cusolver/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cufft/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cusparse/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_curand/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_cupti/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvml/_virtual_includes/headers -Ibazel-out/k8-opt/bin/external/cuda_nvjitlink/_virtual_includes/headers -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/eigen_archive/mkl_include -isystem bazel-out/k8-opt/bin/external/eigen_archive/mkl_include -isystem external/hwloc/hwloc -isystem bazel-out/k8-opt/bin/external/hwloc/hwloc -isystem external/hwloc/include -isystem bazel-out/k8-opt/bin/external/hwloc/include -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src -isystem external/llvm-project/llvm/include -isystem bazel-out/k8-opt/bin/external/llvm-project/llvm/include -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/cuda_cudart/include -isystem bazel-out/k8-opt/bin/external/cuda_cudart/include -isystem external/cuda_cublas/include -isystem bazel-out/k8-opt/bin/external/cuda_cublas/include -isystem external/cuda_cccl/include -isystem bazel-out/k8-opt/bin/external/cuda_cccl/include -isystem external/cuda_nvtx/include -isystem bazel-out/k8-opt/bin/external/cuda_nvtx/include -isystem external/cuda_nvcc/include -isystem bazel-out/k8-opt/bin/external/cuda_nvcc/include -isystem external/cuda_cusolver/include -isystem bazel-out/k8-opt/bin/external/cuda_cusolver/include -isystem external/cuda_cufft/include -isystem bazel-out/k8-opt/bin/external/cuda_cufft/include -isystem external/cuda_cusparse/include -isystem bazel-out/k8-opt/bin/external/cuda_cusparse/include -isystem external/cuda_curand/include -isystem bazel-out/k8-opt/bin/external/cuda_curand/include -isystem external/cuda_cupti/include -isystem bazel-out/k8-opt/bin/external/cuda_cupti/include -isystem external/cuda_nvml/include -isystem bazel-out/k8-opt/bin/external/cuda_nvml/include -isystem external/cuda_nvjitlink/include -isystem bazel-out/k8-opt/bin/external/cuda_nvjitlink/include -fmerge-all-constants -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '--cuda-path=external/cuda_nvcc' '-DMLIR_CAPI_BUILDING_LIBRARY=1' '-DMLIR_CAPI_ENABLE_WINDOWS_DLL_DECLSPEC=1' -DGRPC_BAZEL_BUILD '-Wno-error=cpp' '-std=c++17' -w '-std=c++17' '-DTCP_USER_TIMEOUT=0' -x cuda '-DGOOGLE_CUDA=1' '--no-cuda-include-ptx=all' '--cuda-feature=+ptx83' '--cuda-gpu-arch=sm_60' '--cuda-gpu-arch=sm_70' '--cuda-gpu-arch=sm_80' '--cuda-include-ptx=sm_90' '--cuda-gpu-arch=sm_90' '-Xcuda-fatbinary=--compress-all' '-nvcc_options=expt-relaxed-constexpr' -c external/xla/xla/stream_executor/cuda/all_reduce_kernel_cuda.cc -o bazel-out/k8-opt/bin/external/xla/xla/stream_executor/cuda/_objs/all_reduce_kernel_cuda/all_reduce_kernel_cuda.pic.o)
[23:41:46] # Configuration: 4524842b552e2fd6beb4fe7e1c338e2d4fefde2d4f342cf3e2cf298d856b37f9
[23:41:46] # Execution platform: @@local_execution_config_platform//:platform
[23:41:46] 
[23:41:46] Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
[23:41:46] external/xla/xla/tuple_tree.h:338:36: error: use of undeclared identifier '__T4'
[23:41:46]   338 | (*node) = Node::Leaf(std::forward< __T4> ((pair.second))); 
[23:41:46]       |                                    ^
[23:41:46] 1 error generated.

Working hypotesis is that the file xla/tuple_tree.h is being preprocessed by the CUDA compiler in some weird way, also because the original line 338 of that file is

      *node = Node::Leaf(std::forward<decltype(pair.second)>(pair.second));

which is a bit different from what's shown in the error message, and the mysterious __T4 is mentioned at https://forums.developer.nvidia.com/t/compilation-errors-with-gcc-versions-11-14-and-cuda-toolkit-12-5-12-6-due-to-undefined-builtin-ia32-ldtilecfg-and-builtin-ia32-sttilecfg-etc/308401.