proc-macro crates should build for the host instead of the target

[wildarch]

While experimenting with cross compilation using rules_rust I noticed something odd: it seems like bazel builds crates marked with the proc-macro type for the target platform instead of the host platform. This breaks procedural macros when cross compiling, especially when the target platform does not have a standard library.

I guess that we would want rules_rust to convince bazel to compile procedural macro crates and their dependencies for the host platform instead of the target platform, but I do not know enough about Bazel to determine if and how this would be possible.

For now I came up with an ugly but working solution: I added an extra bool field to rust_library named proc_macro_dep. If this field is set to true, the --target argument is omitted from rustc invocations. I can provide the patch in case anyone is interested.

[mfarrugi]

A pull request with your workaround could be useful to others.

@damienmg do you know if we would be able to dynamically set cfg = "host" in rust_library? The other solution I can come up with is a macro that calls rust_library when the crate isn't a proc_macro, and a new proc_macro rule otherwise.

[wildarch]

One thing we should keep in mind when fixing this is that some crates in the dependency graph might need to be compiled for the target as well as the host if they are both a direct dependency as well as a transitive dependency from a procedural macro. While this would be easy to solve with a duplicate rust_library definition, I think it would be more ergonomic if this was automatically handled by rules_rust.

I will be happy to whip up a PR for this issue once consensus on the right approach has been reached.

[damienmg]

@mfarrugi pulling in @katre and @gregestren who understand how multi-platform configuration works in Bazel.

The main thing is that downstream rust_library should request the proc macro as host, not the target be compiled for host.

@wildarch: how do that works when it needs to be compiled for both? Why would transitive dependencies needs the procedural macro crate?

[mfarrugi]

@damienmg the proc-macro could have dependencies, which then need to be compiled for host, not the other way around.

[wildarch]

What I meant is a case like the following:
We have crates A, B and C.
A and B are normal crates, C is a procedural macro.
If the dependency graph looks like this:

   ---> B
A
   ---> C ---> B

Then we need to build crate B for both the host and the target.

[damienmg]

This is fine, if C is compiled for the execution platform, then their dependencies too. The only thing is that C should be compiled as a host deps.

I don't think this is feasible from the starlark rule context thought. We would have to either declare a different dependencies (which would be weird), or simply do some macro magic although I cannot think of a nice one to do at the moment except if we add a crate_name attribute to rust_library. In that case we would have a rust_host_library with the name of the rust_library and depending on a rust_library_intl with the name "{name}-proc-macro" with a "host" transition. It can then forward it with a provider.

Let's wait until @gregestren chime in and tell us all the magic about dynamic configuration and if there is one that allow us to do that without that last ugly hack.

[gregestren]

@mfarrugi I'm not familiar with Rust - can you clarify more precisely exactly the condition you want to trigger?

i.e. are you asking if you can make a rust_library dynamically set itself to cfg = "host" depending on, say, the value of one of it's attributes? Or if the library's implementation function can dynamically set cfg = "host" for one of its deps?

And what's the specific criteria you want to trigger this on?

[mfarrugi]

A rust_library can be built such that it's a compile-time code generator (similar to a Java annotation processor), and not something that is actually linked into the final output. That's what a proc-macro is. The trigger is a parameter to rust_library, `crate_type = "proc-macro"`. So proc-macro libraries should always be built for the host, but then it's dependencies should also be built for the host (and may need to be built for the target as well).

…

On Thu, Mar 14, 2019, 14:10 Greg ***@***.***> wrote: @mfarrugi <https://github.com/mfarrugi> I'm not familiar with Rust - can you clarify more precisely exactly the condition you want to trigger? i.e. are you asking if you can make a rust_library dynamically set *itself* to cfg = "host" depending on, say, the value of one of it's attributes? Or if the library's implementation function can dynamically set cfg = "host" for one of its deps? And what's the specific criteria you want to trigger this on? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#205 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACLjePld45RS8pWH6a_ZTr3Edrpp234Nks5vWpCjgaJpZM4btrEg> .

[gregestren]

+@juliexxia, @katre

We have the concept of "rule class transitions", which basically lets a rule declare itself as:

rust_library = rule(
   ...,
   cfg = _different_configuration
)

But we don't , I believe, have support for cfg = "host" for that. @katre is doing work that could potentially add that. It's also possible to define _different_configuration to individually set all the flags that host transition would do like --host_cpu, --host_crosstool_top, etc.

Or, if Rust rules fully support platforms I think it'd be possible today to define a transition that sets --platforms= to the host platform? @katre is that realistic or am I missing something?

Alternatively, is it really necessary for both target and host deps to be defined on the same attribute? Can you instead have a binary define deps and host_deps and have the binary's implementation function check that all host_deps have a provider exposing "proc-macro"?

Finally, having a dep set to host, then have one of its dep set back to target might be difficult.

[katre]

It should be possible to use a transition to set the target platform (via the --platforms flag). Saving the previous value and setting it back might or might not work well depending on what other flags and transitions are happening.

[gregestren]

Do rust rules already rely on --platforms?

[wildarch]

They do! Sounds like the way to go 👍

[m3rcuriel]

So I made a quick proof-of-concept that implements the ability to explicitly build crates for host using a 1-1 transition. The problem is that support for attribute-based transitions on rules doesn't exist yet according to the tracking document, but @gregestren can correct me if it's been finished already? With current Bazel features I think we'd have to do it with a separate rule that we can later phase out into just the attribute, or wait for the rust feature.

If we did it with a separate rule we'd have to expose that in cargo-raze as well ¯\(ツ)/¯

[wildarch]

@m3rcuriel Cool! Is your POC available somewhere?