Access to the last used GHC flags

[tseenshe]

This is a question / discussion that can hopefully turn into a well defined feature request, rather than a bug report. I hope it's ok to do it in this format 😅

I am implementing tooling on top of the ghc api (see https://gitlab.com/tseenshe/hsinspect) and I am unfortunately unable to find the correct ghc version and flags to use for any given haskell source file.

I orginally wanted to use the envfiles generated by v2-exec but that 1) requires knowing which flags were last used 2) requires the project to have successfully compiled to get the inplace packages 3) doesn't let me know which ghc flags were used (e.g. lang extensions, preprocessor flags, etc).

I have had a discussion with the cabal-helper author at DanielG/cabal-helper#75 where we concluded that cabal-helper currently doesn't do what I need, and I have had an offline discussion with the hie-bios authors where we concluded that it is not possible to get what I need from the cabal executable by hacking v2-repl (although it's possible to get close, see notes at the end).

I am watching #5954 by @fendor from a distance but I still don't think it gives me what I need, although it might give some useful information.

For any given source file I need to know the ghc version and ghcflags that were last used to compile that file. e.g. if a user is swapping compiler version or optimisation flags, I need access to the most recent. Ideally I'd like to get the information back in a really simple format that doesn't require further processing (e.g. so that I can script it in a launcher script). For example, this UX would be ideal

$ cabal v2-ghcinfo /path/to/file.hs
ghc-version: 8.4.4
ghc-flags: -fbuilding-cabal-package -O0 [...] -optP-DHAVE_TRANSFORMERS -optP-include -optP/home/tseenshe/Projects/ffunctor/dist-newstyle/build/x86_64-linux/ghc-8.4.4/ffunctor-1.2.0/build/autogen/cabal_macros.h -this-unit-id ffunctor-1.2.0-inplace -hide-all-packages -Wmissing-home-modules -no-user-package-db -package-db /home/tseenshe/.cabal/store/ghc-8.4.4/package.db -package-db /home/tseenshe/Projects/ffunctor/dist-newstyle/packagedb/ghc-8.4.4 -package-db /home/tseenshe/Projects/ffunctor/dist-newstyle/build/x86_64-linux/ghc-8.4.4/ffunctor-1.2.0/package.conf.inplace -package-id base-4.11.1.0 -package-id transformers-0.5.5.0 -XHaskell2010 Data.FFunctor -Wall -Werror=missing-home-modules -hide-all-packages -ferror-spans

If there is no cache hit (e.g. if the user hasn't called v2-build or v2-run yet), then it is ok to exit with a failure.

Appendix 1: hie-bios uses a hack that ultimately doesn't work. It uses a wrapper script to masquerade as ghc. But 1) there is no way to find out which parameters the user last used 2) the cache will always miss because the ghc hash will change.

Appendix 2: I would love it if we got into trading features like old fashioned hippies. e.g. if you implement this feature in cabal-install, I will prioritise your feature request in haskell-tng/hsinspect.

[fendor]

Does #5954 not give you the information you require?
Also, different to your given requirements, show-build-info establishes a project context, e.g. it configures a project if it isnt configured yet. In this sense, I think it would give the latest information, however, it does not seem to be fast.
What you would require is another output format, do I see that correctly? Or is more information missing?
Generally, I would be happy to keep hacking on it after #5954 has finally been merged.

[tseenshe]

@fendor I think build-info provides a bunch of useful information that I can use for other kinds of tooling. e.g. I could use the component name mappings to generate commands and I can use the source directories (combined with component names) to automate the generation of test files.

But for this particular usecase of programmatically using ghc, it doesn't do what I need. In particular

1. it doesn't use the latest user settings, it still requires me to infer them from the user and/or provide them to the command.
2. I don't ever want to trigger a lengthy Resolving job (or compilation) I'd much rather get an error
3. it provides complex data structures that I'd need to parse, I just want output that I can reuse verbatim without having to know what it means.
4. I need it to be fast, maximum 1 second. I can cache to a certain extent but that always has flaws.

[DanielG]

@fendor Maybe these requirements bring us back to actually generating a build.json as part of the build like @hvr suggested I think.

That would give the sort of last-used info @tseenshe wants without having to incur any build side-effects.

[tseenshe]

@DanielG I'm not completely opposed to having to parse JSON, although I'd rather if there was no parsing involved (I have to do the parsing in the editor and there are many reasons why JSON parsing is a PITA there). But I would request that at least one of the fields is the full set of ghc flags so I don't have to regenerate it from different sections.

A workaround I was considering was having a compiler plugin dump the compiler version and flags to a hidden file at the base of the source directory.

[DanielG]

I don't think it would be a problem to also provide a slimmed down version for easy parsing as long as you don't expect it to mirror all the info in build.json exactly. Alternatively we can also dump build.sexp with everything, if your editor can't parse S-expressions it's broken IMO :P

One thing about your requirements I don't really understand is why you're so adamant about getting exactly the options of the last invocation. The way the workflow is supposed to work is that you use v2-configure to make any options you want cabal to respect by default permanent.

As a user of your tool I might actually be slightly irritated about it using whatever I used last to run v2-build. What if I have some one-off thing I want to run through v2-build that I don't want to use for normal builds? I dunno, maybe I want to quickly see if ghc-7.10 still works with v2-build -w ghc-7.10 but not have that be persisted because 8.6 has better wanings or run some crazy ghc plugin that just does some analysis and doesn't produce sensible build output.

[tseenshe]

I don't think it would be a problem to also provide a slimmed down version for easy parsing

If it's just the compiler and flags, that'd good enough for me!

if your editor can't parse S-expressions it's broken IMO :P

🤣 totally agree, but unfortunately I want the unenlightened to be able to use my tool as well. I'm ok parsing JSON from emacs for additional (optional) feature sets, so no need for s-expression output.

BTW there is one more hack that @rahulmutt suggested, which is to write a binary parser for the extrapolated-plan. This could allow for backwards compat and immediate use until cabal-install supports dumping out the build plan (including the ghc flags).

As a user of your tool I might actually be slightly irritated about it using whatever I used last to run v2-build

I would be very surprised if the example you can come up with was a common situation. It can be easily worked around by issuing a command using your regular compiler version. One of the overriding design constraints of haskell-tng and hsinspect is to work for 99% of the time and to make it easy to correct the remaining 1%. Working around that final 1% is a rabbit hole I am not interested in (unless somebody wants to pay me).

Aside: the cache miss cost is so high on larger projects that I'd personally like cabal to require an additional flag so that I have to confirm "yes I really mean for you to wipe out the resolution cache that takes 60 seconds to create".

[tseenshe]

he way the workflow is supposed to work is that you use v2-configure to make any options you want cabal to respect by default permanent.

That's not the same thing. That doesn't account for local flags such as --enable-tests (or benchmarks) or enabling/disabling profiling.

[DanielG]

if your editor can't parse S-expressions it's broken IMO :P

totally agree, but unfortunately I want the unenlightened to be able to use my tool as well. I'm ok parsing JSON from emacs for additional (optional) feature sets, so no need for s-expression output. The core feaures should be zero dependency.

I didn't really mean just anything that's not emacs, but s-expressions really are super easy to parse and I bet you can find a 20loc parser for any editor you want to support but honestly the same can be said of json.

As a user of your tool I might actually be slightly irritated about it using whatever I used last to run v2-build

I would be very surprised if the example you can come up with was a common situation. It can be easily worked around by issuing a command using your regular compiler version.

That's fair just so long as the user is informed about whats going on and that it's different from the default behavior of the build-tool.

One of the overriding design constraints of haskell-tng and hsinspect is to work for 99% of the time and to make it easy to correct the remaining 1%.

Haha, that's funny. I started working on ghc-mod and cabal-helper and all that because I was pissed about how tools only used to work in like 80% of cases and vowed to make them work 100% of the time. Well 95% so far but trending upwards ;)

Aside: the cache miss cost is so high on larger projects that I'd personally like cabal to require an additional flag so that I have to confirm "yes I really mean for you to wipe out the resolution cache that takes 60 seconds to create".

I'll be waiting for your PR to add a --no-run-solver option :) Honestly I don't think that would be very hard to add at all. Personally I don't have projects that take that long to solve so I don't see the need but if you have that use-case you should solve it.

[DanielG]

The way the workflow is supposed to work is that you use v2-configure to make any options you want cabal to respect by default permanent.

That's not the same thing. That doesn't account for local flags such as --enable-tests (or benchmarks) or enabling/disabling profiling.

Yes it is you can pass any option to v2-configure that you can pass to v2-build (well all the things you mention anyways) or what do you mean?

[tseenshe]

@DanielG until cabal.project.local has support for flags that only apply to local projects (not even source deps) then I don't think this is workable. e.g. tests: True is not workable at present and is much better to pass as a command line argument. Wo betide those who forget it, even once, for the solver will miss the cache and do it all again...

[DanielG]

Ah I see what you mean. Actually cabal.project.local has support for per-package flags like so:

package cabal-helper
  tests: True

That used to be my workaround for flags applying to the entire build-plan anyways. I don't think you can make that happen from the cmdline though.

The other thing is I think the behaviour where v2-configure flags would apply to the entire build-plan and not just the local packages was fixed around 2.0.*. I remember having that problem in the old days too but I've never noticed it again so I think that was fixed.

[tseenshe]

I have a (very slow) workaround to this problem that uses a heavily modified version of the hie-bios hack

https://gitlab.com/tseenshe/haskell-tng.el/blob/tng/cabal-ghcflags.sh

This dumps out .ghc.flags files at the root of every source directory in the project (for all components).

It's really really slow, and it doesn't support when the user changes their compilation preferences, and it requires the user to keep it in sync with the build definition.

If cabal generated these files automatically (instead of the .ghc.env.* files) then it would be incredibly useful for tooling authors like me.