Provide middle-ground overlay between pkgsMusl and pkgsStatic

[nh2]

Issue description

For my work on static-haskell-nix I need an overlay that has both .so files and .a files.

This is because I need .as for linking Haskell exes statically, but .so files for executing TemplateHaskell.

Right now pkgsStatic not only adds .a files, but also disables .so files:

• makeStaticBinaries and makeStaticLibraries set --disable-shared
• static.nix uses those
• static.nix disables shared Haskell libs that TH needs

Proposal

I think we should have the following stack of overlays:

pkgsMusl <- pkgsStatic <- pkgsStaticOnly

• pkgsMusl is what it is now: All packages using musl libc
• pkgsStatic should be the one that has .a files added to all libraries (in addition to .so files), and all binaries (including Haskell ones) linked statically where possible
• pkgsStaticOnly should be what pkgsStatic is now: Having only static libraries and binaries, getting rid of .so files.

---

CC @matthewbauer @Ericson2314 @dtzWill @nmattia

[Ericson2314]

So what's the deal with static libs and GHCi/TH? It used to work? CC @angerman

[matthewbauer]

Yeah I was hoping we could use -fexternal-interpreter for this case.

[nh2]

Related: I asked on #ghc:

Can ghc currently load static libs when evaluating TH, e.g. libgmp.a? My musl based builds fail in pkgsStatic when I don't have libgmp.so

@angerman said:

yes it can. If you disable the DYNAMIC define.
there is a call called loadDLL or something that just defaults to shared objects unless ghc is explicitly set to not use dynamic libraries.
angerman
https://github.com/ghc/ghc/blob/610ec224a49e092c802a336570fd9613ea15ef3c/compiler/ghci/Linker.hs#L1467

@cocreature said:

angerman: that line only applies to Haskell libraries (which funnily enough is why you can get GHC to load static Haskell libraries in TH by putting them in extra-libraries instead of hs-libraries) so it shouldn’t be relevant for libgmp

we actually use the extra-libraries trick at work (for various reason that are probably better explained at ZuriHac :)) and surprisingly it hasn’t exploded so far

the logic for C libraries should fall back to static libs afaik so not sure what issue nh2 is seeing

it looks like if findArchive fails it will fall into the assumeDll so maybe GHC doesn’t find the libgmp.a that nh2 has lying around?

[matthewbauer]

My main concern with building both dynamic and static is that build systems will get messed up when they see both. They might try to use the wrong one and do the wrong thing. It's better they fail than do the wrong thing here. It's very similar to why we don't build multi-target things as well. Nix means "nothing", not everything 😸 .

[nh2]

build systems will get messed up when they see both

@matthewbauer In what situation does that happen though? Most build systems have a flag that controls whether you want static or dynamic exes. If you link with -static, it's going to use .a files, or fail complaining (or at least that's what I've observed so far).

pkgsStaticOnly would still be as you desire, but having the middle thing as well does allow us to build non-cross static executables that need .sos on the way there quite effortlessly today.

[Ericson2314]

So from the GHC quotes this does sound quite fixable?

[angerman]

So from the GHC quotes this does sound quite fixable?

Everything is fixable if you apply enough WD40 and/or duct tape.

[nh2]

So from the GHC quotes this does sound quite fixable?

@Ericson2314 It is possible, but what I'm looking for is a way to get static Linux builds immediately using only as-standard-as-possible approaches without teaching GHC new things. The easiest way to get there is to have .so + .a. Many projects already benefit from distributing static binaries this way, including dhall, stack and others.

That is not to say there's no merit in also working on the other approaches, but because these things have users already right now, I'd like to move them to nixpkgs rather sooner than later, because maintaining all the C library overrides to add .as in static-haskell-nix is an effort that would go away if we could share those overrides with pkgsStatic.

[Ericson2314]

@nh2 I'm mainly just worried about arbitrary non-Haskell packages misbehaving when both are available. When I say it sounds easy to change GHC, I mean easy to write a patch and backport it to use in nixpkgs immediately. Based only my experience working on Cabal in the past, and working on GHC now, I'd say landing GHC patches is easier in general too, lest your experience with the Cabal patch makes it seem that any upstream change is invariably a chore.

[nh2]

I'm mainly just worried about arbitrary non-Haskell packages misbehaving when both are available.

@Ericson2314 Do you have an example? What kind of scenario do you have in mind?

When I say it sounds easy to change GHC, I mean easy to write a patch and backport it to use in nixpkgs immediately.

I don't doubt that we can do thinkgs quickly in both GHC and nixpkgs, but I'm unclear on what exactly we need to do on the GHC side to make it never look at .so files, and who can work on that in the immediate future.

In general, is there a drawback to what I propose (having one place / adapter / overlay that simply enables .as for as much as possible)? The concerns you may have about packages misbehaving would not apply to what's now pkgsStatic, as that one would be unmodified (turning .sos off).

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.