RBE Failures related to protobuf, abseil-cpp, and `#include <filesystem>`

[dougthor42]

RBE failures

🐞 bug report

Affected Rule

N/A

Is this a regression?

Hard to track down - recent RBE CI jobs in PRs have failed. The latest commit to main (#3021) also failed but I can't think of any way those changes would break RBE...

Description

RBE CI jobs are failing; multiple instances across mostly independent PRs:

• fix(gazelle) Update gazelle to properly process multi-line python imports #3077: https://buildkite.com/bazel/rules-python-python/builds/12151/steps/canvas?sid=0197f676-75cc-4487-ade9-de0fd0ca32f4
• feat(gazelle): Gazelle plugin generates py_proto_library #3057: https://buildkite.com/bazel/rules-python-python/builds/12172/steps/canvas?sid=01980196-de9e-413b-8249-c0dc3c3fb983
• feat(gazelle): Add include_pytest_conftest annotation #3080: https://buildkite.com/bazel/rules-python-python/builds/12160/steps/canvas?sid=0197fbe6-8c0d-4f96-bfbf-fd2b225b9c4f

The common theme across PRs is that they're all related to the Gazelle plugin.

However, other Gazelle PRs made it through just fine:

• chore: Switch back to smacker/go-tree-sitter #3069
• docs: Typo in gazelle/README.md #3040
• fix(gazelle) Register pyi_deps as ResolveAttrs #3037

🔬 Minimal Reproduction

N/A

🔥 Exception or Error

Buildkite logs can't show everything - you have to download the log file. Here's an excerpt:

bk;t=1752373435607^G(02:23:55) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/protobuf~/src/google/protobuf/compiler/java/lite/BUILD.bazel:1:11: Compiling src/google/protobuf/compiler/java/lite/make_field_gens.cc [for tool] failed: (Exit 1): gcc failed: error executing CppCompile command (from target
 @@protobuf~//src/google/protobuf/compiler/java/lite:field_generators) 
bk;t=1752373435607^G  (cd /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/execroot/_main && \
bk;t=1752373435607^G  exec env - \
bk;t=1752373435607^G    PATH=/bin:/usr/bin:/usr/local/bin \
bk;t=1752373435607^G    PWD=/proc/self/cwd \
bk;t=1752373435607^G  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++17' -MD -MF bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/java/lite/_objs/field_generators/make_fie
ld_gens.d '-frandom-seed=bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/java/lite/_objs/field_generators/make_field_gens.o' -iquote external/protobuf~ -iquote bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~ -iquote external/abseil-cpp~ -iquote bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/abseil-cpp~ -iquote external/
zlib~ -iquote bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/zlib~ -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/java/lite/_virtual_includes/field_generators -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes/protobuf -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~
/src/google/protobuf/_virtual_includes/internal_visibility -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes/port -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes/protobuf_lite -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes
/arena -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes/arena_align -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/stubs/_virtual_includes/lite -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes/arena_allocation_policy -Ibazel-out/k8-opt-exec
-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes/arena_cleanup -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes/string_block -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/_virtual_includes/endian -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src
/google/protobuf/_virtual_includes/varint_shuffle -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/io/_virtual_includes/io -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/io/_virtual_includes/io_win32 -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/third_party/utf8_range/_virtual_includes/utf8_v
alidity -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/third_party/utf8_range/_virtual_includes/utf8_range -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/io/_virtual_includes/gzip_stream -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/stubs/_virtual_includes/stubs -Ibazel-out/k8-opt-exec-ST-1
50d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/io/_virtual_includes/printer -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/io/_virtual_includes/zero_copy_sink -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/io/_virtual_includes/tokenizer -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/
src/google/protobuf/compiler/java/_virtual_includes/generator_common -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/java/_virtual_includes/helpers -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/_virtual_includes/code_generator -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/
src/google/protobuf/compiler/_virtual_includes/code_generator_lite -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/_virtual_includes/retention -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/_virtual_includes/versions -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/
protobuf/compiler/java/_virtual_includes/internal_helpers -Ibazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/java/_virtual_includes/java_features_bootstrap -isystem external/zlib~/zlib/include -isystem bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/zlib~/zlib/include -g0 -g0 -c external/protobuf~/src/google/protobuf/compiler/java/lit
e/make_field_gens.cc -o bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/java/lite/_objs/field_generators/make_field_gens.o -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"')
bk;t=1752373435607^G# Configuration: b9113c5f6273d63b11251bd220fa9e82b592016cf064716f09cbcb27749c2db2
bk;t=1752373435607^G# Execution platform: @@_main~internal_dev_deps~buildkite_config//config:platform
bk;t=1752373435607^GIn file included from external/abseil-cpp~/absl/hash/hash.h:85:0,
bk;t=1752373435607^G                 from external/abseil-cpp~/absl/container/internal/hash_function_defaults.h:55,
bk;t=1752373435607^G                 from external/abseil-cpp~/absl/container/flat_hash_map.h:41,
bk;t=1752373435607^G                 from bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/java/_virtual_includes/helpers/google/protobuf/compiler/java/context.h:14,
bk;t=1752373435607^G                 from bazel-out/k8-opt-exec-ST-150d2d5f4ddd/bin/external/protobuf~/src/google/protobuf/compiler/java/lite/_virtual_includes/field_generators/google/protobuf/compiler/java/lite/make_field_gens.h:14,
bk;t=1752373435607^G                 from external/protobuf~/src/google/protobuf/compiler/java/lite/make_field_gens.cc:12:
bk;t=1752373435607^Gexternal/abseil-cpp~/absl/hash/internal/hash.h:66:10: fatal error: filesystem: No such file or directory
bk;t=1752373435607^G #include <filesystem>  // NOLINT
bk;t=1752373435607^G          ^~~~~~~~~~~~
bk;t=1752373435607^Gcompilation terminated.

🌍 Your Environment

Operating System:

N/A

Output of bazel version:

N/A

Rules_python version:

N/A

Anything else relevant?

[aignas]

Created #3083 to see if it is Ubuntu that is too old but it does not seem the case.

@meteorcloudy, could you help with trying to debug this?

[dougthor42]

Is our BuildKite config and RBE config completely stored in the repo? Or is there, for example, a web frontend to BuildKite where settings and config is made?

It doesn't look like the config is fully stored in rules_python, but maybe I'm missing things. I see .bazelci/presubmit.yml which is job config and has things like platform: rbe_ubuntu2004 or platform: windows, but I don't see things like OCI image definitions (Dockerfile) for rbe_ubuntu2004, compute resource machine sizes (vCPU/RAM), or any of the Bazel RBE config like --remote_executor, --remote_instance, or --remote_cache.

[rickeylev]

.bazelci/presubmit.yml is the job config. The buildkite machines are managed by bazelci.

The RBE stuff comes from an external repo. I think @bazelci_rules, which has some rbe_config thing.

[dougthor42]

Thanks Richard. Sounds like it's not something I'll really be able to assist with, so I'll leave it to Yun and Ignas.