Support --incompatible_enable_proto_toolchain_resolution

[kusaeva]

Is your feature request related to a problem?

The problem is need to compile protoc instead of using prebuilt binary while building java_grpc_library

Describe the solution you'd like

Support --incompatible_enable_proto_toolchain_resolution bazel flag so users can override protoc compiled from sources with prebuilt binary, as already done in rules_python or WIP in rules_go

Describe alternatives you've considered

Additional context

Examples of declarations toolchains for other languages that already support the feature here

[kannanjgithub]

Feel free to raise a PR for this.

[mortenmj]

@kannanjgithub I had a look at what it would take to make a PR for this, and there are two parts.

One is using a toolchain to provide protoc, to avoid the dependency on @com_google_protobuf//:protoc. That is pretty straight forward. I could make a separate PR for that, as it would be pretty small and easy to review.

The second part involves the dependency on @com_google_protobuf//:protoc_lib, which I don't think is very easy to avoid. If we make it easy for the user to declare their own java_rpc_toolchain, users could opt to consume prebuilt binaries from maven while still maintaining the existing default of building from source. Does that sound like something to move ahead with?

[alexeagle]

does grpc/grpc#41274 cover this, or is a change needed in this repo too?

[kannanjgithub]

In java_grpc_library bzl rule definition we use the below toolchain

toolchains = ["@bazel_tools//tools/jdk:toolchain_type"],

To use the protoc toolchain to use prebuilt binary I think we will have to include that tool chain like

toolchains = toolchains.use_toolchain(toolchains.PROTO_TOOLCHAIN),

done here for example.

I'm not sure about the @com_google_protobuf//:protoc_lib point that @mortenmj raised though.

[ejona86]

@alexeagle, this repo has no relationship with grpc/grpc as far as Bazel is concerned. That change looks worrisome; surely we aren't all expected to use a "private" API. When protobuf thinks it is ready, they need to make it public. Before then, it doesn't seem things are in-place. Creating our own toolchain would technically work, but then that becomes its own separate thing that has to be migrated to be shared with protobuf once protobuf is ready.

[ouillie]

That change looks worrisome; surely we aren't all expected to use a "private" API.

I assume this refers to the toolchain_helpers.bzl library that I included in grpc/grpc#41274.

Perhaps @alexeagle can speak to this with more authority, but I wouldn't call it "using a private API". In general, private APIs are worrisome because upstream changes could break downstream dependents, but that risk does not apply to this sort of vendored helper library. The "API", which to my knowledge is considered production ready, consists entirely of the --incompatible_enable_proto_toolchain_resolution flag and the protoc toolchain type *. The helper library just makes using those APIs more convenient. I believe they didn't want to make the helper library public because that would make removing it in the future much more complicated, even after the Protobuf module itself stops using the library.

The new, optional prefer_prebuilt_protoc flag is sort-of orthogonal to the API here. It just provides implementations of the toolchain for various common platforms. You can always provide your own implementations if you want.

* Unfortunately, the toolchain type is defined in a package called private. I can see that being a concern. Perhaps it's worth moving that to a new package and marking it as stable, although I'm not sure if moving it is even practical at this point.

[alexeagle]

Yup, the toolchain_helpers.bzl could either be considered a "friend" visibility, given your relationship with the protobuf team, or you just vendor that file into the repo. The latter approach doesn't risk any version drift as this was a one-time migration.

[ejona86]

proto_toolchain_helpers.bzl has references to load("@com_google_protobuf//bazel/private:native.bzl", "native_proto_common") and @com_google_protobuf//bazel/private:proto_toolchain_type.