Add Homebrew libraries to the library search on Apple Silicon/darwin-arm64, or how to find them from other software?

[eregon]

Provide a detailed description of the proposed feature

I wonder why Homebrew does not seem to add the libraries it installs to the system library search path on Apple Silicon/darwin-arm64.
Specifically, the case I care about here is Ruby FFI and other dlopen() users, and how they are supposed to find libraries installed via Homebrew: ffi/ffi#880 (comment)
I'm also interested in the general question: "How should software compiled from source (e.g., ruby) find Homebrew-installed headers+libraries?"

There is some discussion in #9177 and notably #9177 (comment)
But I am not clear on what's really a solution there.

• brew --prefix wget is unusable for FFI purposes as we only get the library name, no the package name.
• brew shellenv does not seem applicable as we need to stay in the same process

What is the motivation for the feature?

Make it easier for Ruby FFI and other dlopen() users to use Homebrew-installed libraries.

How will the feature be relevant to at least 90% of Homebrew users?

By making it easier (ideally automatic with no user intervention) for Ruby FFI and other dlopen() users to use Homebrew-installed libraries (and even software compiled from source in general) to find Homebrew packages, there is less need to know about lots of details (e.g., how to pass the openssl dir for ruby, actually not trivial it's --with-openssl-dir + PKG_CONFIG_PATH) and workarounds to make it work.
My observation is many macOS users in particular seem to not know much about their system, so the less they need to tweak the better chance they do not mess it up (which seems to happen a lot given a lot more report from macOS rather than Linux on most softwares).

What alternatives to the feature have been considered?

Searching a hardcoded directory like /opt/homebrew/lib/ but it feels like a ugly workaround and might not work long-term, plus it does not scale to maintain such exceptions for every platform.
Another way to think about this is every single platform supported by Ruby FFI needs no special search path, except darwin-arm64, how can we avoid it?

[eregon]

There was some discussion in https://github.com/orgs/Homebrew/discussions/1600 related to this, cc @larskanis, but it does not seem to lead to a clear or official solution to me.

[MikeMcQuaid]

Searching a hardcoded directory like /opt/homebrew/lib/ but it feels like a ugly workaround and might not work long-term, plus it does not scale to maintain such exceptions for every platform.

Shell out to brew --prefix and search $(brew --prefix)/lib. Alternatively, hardcode Homebrew's default directories. I don't really see other options here, unfortunately.

[eregon]

Thanks for the quick reply.
To understand better, would it make sense to add /opt/homebrew/lib to e.g. DYLD_FALLBACK_LIBRARY_PATH?
Or Homebrew on darwin-arm64 wants on purpose to be found by no build system/dlopen() unless added explicitly? (to avoid potential conflicts?)

[MikeMcQuaid]

To understand better, would it make sense to add /opt/homebrew/lib to e.g. DYLD_FALLBACK_LIBRARY_PATH?

By Homebrew or by FFI? I'd say "probably" for us and "maybe" for you 😁

We'd need to set it in brew shellenv and I'm not convinced that:

• this is used widely enough to be useful
• this would have no other negative effects

Or Homebrew on darwin-arm64 wants on purpose to be found by no build system/dlopen() unless added explicitly? (to avoid potential conflicts?)

No, this is not intentional.

[eregon]

I meant by Homebrew, but maybe it's interesting to see if that'd work for FFI. I'm not sure that modifications to this env var are considered after the process started, would need to check.

No, this is not intentional.

Interesting, then I think it'd be worth considering if setting DYLD_FALLBACK_LIBRARY_PATH and/or other env vars to make packages installed by Homebrew available by default would make sense (e.g., PATH is already set).
Making Homebrew packages available by default already worked on darwin-amd64 without needing env vars, due to being in a standard /usr/local prefix which is already considered by compilers/linkers/etc, so in some sense there is some regression here on darwin-arm64 in that it's much harder to use Homebrew packages for compiled-from-source software and dlopen().

[MikeMcQuaid]

Interesting, then I think it'd be worth considering if setting DYLD_FALLBACK_LIBRARY_PATH and/or other env vars to make packages installed by Homebrew available by default would make sense (e.g., PATH is already set).

Yup. @Homebrew/maintainers any thoughts here?

I don't think we can rely on brew shellenv alone, here. We'd probably also need a brew doctor check.

Making Homebrew packages available by default already worked on darwin-amd64 without needing env vars, due to being in a standard /usr/local prefix which is already considered by compilers/linkers/etc, so in some sense there is some regression here on darwin-arm64 in that it's much harder to use Homebrew packages for compiled-from-source software and dlopen().

That's fair 👍🏻

[eregon]

this is used widely enough to be useful

Probably a bit of a chicken-and-egg problem, i.e., if brew shellenv would set everything needed to use Homebrew packages then it'd probably see a lot more usage.

this would have no other negative effects

Yes, definitely something to consider, and should system packages be in front or behind Homebrew packages, should a package/library/header exists for both.

[MikeMcQuaid]

Probably a bit of a chicken-and-egg problem, i.e., if brew shellenv would set everything needed to use Homebrew packages then it'd probably see a lot more usage.

Sorry, I don't agree here. I'm thinking partly "people who don't need brew shellenv to set their environment shouldn't be forced to do so" and also "things I run through e.g. VSCode may not run brew shellenv".

[fxcoudert]

Be aware that there are some security restrictions to the use of DYLD_LIBRARY_PATH:

Any dynamic linker (dyld) environment variables, such as DYLD_LIBRARY_PATH, are purged when launching protected processes.

[carlocab]

Not a fan of DYLD_[FALLBACK_]LIBRARY_PATH, and neither is Apple:

❯ FOO=foo /bin/bash -c 'echo ${FOO-unset}'
foo
❯ DYLD_LIBRARY_PATH=foo /bin/bash -c 'echo ${DYLD_LIBRARY_PATH-unset}'
unset
❯ DYLD_FALLBACK_LIBRARY_PATH=foo /bin/bash -c 'echo ${DYLD_FALLBACK_LIBRARY_PATH-unset}'
unset

This is basically the security restriction that @fxcoudert references above.

I am open to adding $HOMEBREW_PREFIX/lib to binary RPATH entries as needed to specific formulae. This will allow those formulae to call dlopen on libraries that are linked into $HOMEBREW_PREFIX/lib. It doesn't help with things users have built themselves, but I suppose we can add documentation suggesting adding the necessary RPATH entries wherever needed.