Pass *-options to GHC when compiling ordinary Haskell sources

[zlonast]

Will close issues #9801 #4435
Main idea #9801 (comment)

IMHO, cc-options (and ld-options and cpp-options and ...) should be always passed when invoking GHC, similarly as ghc-options should be always used when invoking ghc - regardless of what is the intention of a particular GHC-call. GHC might use or not use the options, Cabal cannot know and should not guess.

cc-options, cxx-options, jspp-options, asm-options, cmm-options, ld-options, cpp-options,
should be always passed when invoking GHC, similarly as ghc-options should be always used when
invoking ghc - regardless of what is the intention of a particular GHC-call.
GHC might use or not use the options, Cabal cannot know and should not guess.

cc-options and cxx-options still need to be separated (C/C++) for versions below 8.10.
https://gitlab.haskell.org/ghc/ghc/-/issues/16477

---

Template Α: This PR modifies behaviour or interface

Include the following checklist in your PR:

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
  • Is the change significant? If so, remember to add significance: significant in the changelog file.
• The documentation has been updated, if necessary.
• Manual QA notes have been included.
• Tests have been added. (Ask for help if you don’t know how to write them! Ask for an exemption if tests are too complex for too little coverage!)

---

Depends-on: #11711
Depends-on: #11713

[zlonast]

I probably need to think about tests, maybe something started working besides CApiFFI 🤔

[zlonast]

Ok, this only works for things newer than 8.8
gcc: fatal error: cannot specify ‘-o’ with ‘-c’, ‘-S’ or ‘-E’ with multiple files

Ok, got it, we just need to separate the flags -optc and -optcxx
Pass -optcxx for GHC >= 8.10 #7072

[zlonast]

@ulysses4ever @phadej It's not very easy to add flags to sources files due to backward compatibility, so for now I suggest considering the pull request as is ¯\_(ツ)_/¯ (Anyway, this solved the problem with macros in header files)

[ulysses4ever]

Great start, thank you! Below are some inline questions.

One general question: did you check that this doesn't lead to duplicated options when cabal calls GHC? I assume other *-options are passed today sometimes. So, sometimes we'll get those options passed as today and through your change too. Is that right? It seems undesirable.

It's not very easy to add flags to sources files due to backward compatibility

Can you be more specific? Do you mean it's hard to test on older GHCs? This shouldn't be a problem because the tests can run for specific versions of GHC (e.g. starting from 8.10).

[geekosaur]

Same concerns as @ulysses4ever raised.

[phadej]

did you check that this doesn't lead to duplicated options when cabal calls GHC?

Yes, it looks like so.

I'd say the problem is rather that GhcOptions is not used uniformly. There is e.g. componentCcGhcOptions which does set ghcOptCcOptions. Why these don't show up when compiling files CApiFFI headers?

IMHO, it's wrong that there are separate componentAsmGhcOptions, componentJsGhcOptions, componentGhcOptions. There should be just one componentGhcOptions with all the options, and then it's up to GHC which ones are relevant, and which aren't.

EDIT: There might be situations where ghcOptMode or something like that is different, and input files "obviously". But the bulk of options should be the same. Starting with verbosity. I'd imagine there is only single line in the codebase which does ghcOptVerbosity = toFlag (min verbosity normal). Currently there are six copies of that. Cannot be right. I suspect there are many subtle inconsistencies hiding because of that.

EDIT: So AFAICT, this is not the correct fix, not what I'd imagine seeing.

[ulysses4ever]

@zlonast could you comment on this #10969 (comment)? Specifically, two things: was the issue resolved and what you think of trying a large scale compilation (perhaps stackage, maybe via clc-stackage).

[zlonast]

@zlonast could you comment on this #10969 (comment)? Specifically, two things: was the issue resolved and what you think of trying a large scale compilation (perhaps stackage, maybe via clc-stackage).

The problem wasn't my fault initially. (A bug in the mac os clc-stackage blacklist.)

Yes, it's been resolved.

The last test run was successful. (on linux and mac os )

I suppose, we should run another one?

[ulysses4ever]

The last test run was successful. (on linux and mac os )

Is it something you ran on your side? If there was a successful clc-stackage run at some point, I don't see a reason to re-run it. Over to you @mpickering .

[ulysses4ever]

@zlonast GitHub UI says the branch is out of date. Please rebase when you can, or, if you prefer, I can do it (after all, it's an endless game before the merge happens, and we don't want contributors to feel like they got an indefinite assignment for it).

[zlonast]

The last test run was successful. (on linux and mac os )

Is it something you ran on your side? If there was a successful clc-stackage run at some point, I don't see a reason to re-run it. Over to you @mpickering .

Yes, I tried it on my side, I would be glad if someone could do it too.

I recently rebased, so feel free to do so with my branch.

[geekosaur]

There are a lot of unresolved comments still, that will need to be resolved to merge.

[geekosaur]

FWIW I'm not worried about rebases, except insofar as they may help with development; since PRs are being merged all the time, it's usually not worth the resulting constant rebases when Mergify will do it itself.

[ulysses4ever]

@zlonast please, attend to the inline comments when you get a chance

[zlonast]

I've closed a few threads because I don't think they're that important, but feel free to reopen them.

phadej
Ok, I see. The GHC-8.8 doesn't know about C++ compiler, so we have to deal with it separately, and in an ugly way.

Could you write a note, referencing the test. By looking at the code only, it looks wrong or at least suspicious.

In the retrospective, cabal-install could refused to support an accidental "but it did work before with older GHC, cc-options were passed to C compiler" when proper cxx-options were added. Those accidents just make code base very spagetti.

zlonast
in some reality it is true, but due to problems with linking on versions lower than 9.4, I just don't know what to do

https://gitlab.haskell.org/ghc/ghc/-/issues/17919

I think this is the main part from which the problems grow, the issue itself is about dynamic linking, but I could not find where and how the static one was fixed

[zlonast]

Perhaps we should add more tests, but I don't have the energy for that right now.

ulysses4ever
Could you, at least, check that the test suite has tests for all the other-language options (ASM, JS, CXX, CMM, LD)? And if some of these options don't have tests, could you add something simple for them? I agree that it's not fun to do this exercise, but I think it's a big change and I want at least a minimal guarantee that we're not breaking things badly.

zlonast
cc-options - cabal-testsuite/PackageTests/FFI/ForeignOptsC/README.md
cxx-options - cabal-testsuite/PackageTests/FFI/ForeignOptsCxx/README.md
jspp-options - cabal-testsuite/PackageTests/JS/JSPPOptions/README.md
asm-options - none
cmm-options - cabal-testsuite/PackageTests/Cmm/CmmSources/README.md
ld-options - ParserTests
cpp-options - ParserTests
-pgmc - cabal-testsuite/PackageTests/ShowBuildInfo/Complex
-O1 and -O2 for C++ and C and Asm - cabal-testsuite/PackageTests/ShowBuildInfo/Complex

if it still reproduces we should start tests for asm-options and -pgmc/-pgma #6534

[zlonast]

There are some linking issues with older versions. I don't know what to say about that.

      , -- there are problems with linking with versions below 9.4,
       -- that's why we need this replacement for linkGhcOptions
       -- see example in cabal-testsuite/PackageTests/ShowBuildInfo/Complex
       ghcOptCcProgram = Internal.defaultGhcOptCcProgram lbi

mpickering
What "problems with linking" and what exact issue does this solve? It's quite mysterious at the moment.

phadej
Agree, these kinds of comments should include links to GHC issues

This is not https://gitlab.haskell.org/ghc/ghc/-/issues/17919, as that one is pre GHC-8.10

zlonast
I still don't know where it was fixed, my search methods were weak. I only know that it happened with the release 9.4.*

[zlonast]

And also we have problems with normalizer. #11187 (comment)

zlonast
the real problem is that when adding -pgm* flags we need to write at least 5(10) tests instead of one because the output contains the path to the C compiler

1. no -pgmc
single-2.out
single-2.test.hs
2. gcc is /usr/bin/gcc
single.out
single.test.hs
3. clang is /tools/ghc-<GHCVER>/lib/../mingw/bin/clang
win-single-4.out
win-single-4.test.hs
4. clang is /ghcup/ghc/<GHCVER>/lib/../mingw/bin/clang
win-single-8-6.out
win-single-8-6.test.hs
5. clang is /ghcup/ghc/<GHCVER>/lib/../mingw//bin/clang
win-single.out
win-single.test.hs

geekosaur
That sounds more like a bug / deficiency in the normalizer to me.

[Mikolaj]

I see the diff is quite large and there is scope creep (possible interactions, hypothetical scenarios, related buglets), so I'm worried for the kind PR author. Could we all try to help somehow? Maybe let's try to answer some questions @zlonast poses? Could we somehow split responsibility for some subtasks or open new tickets for them and leave them undone for now? It would be really great to merge this PR while there still the momentum.

[ulysses4ever]

The only unresolved thread that I was able to find in this (rather lengthy) discussion is this: #10969 (comment) started by @mpickering, so, maybe Matt could try to push it forward (or backward :-))

[Mikolaj]

Sadly, @mpickering may not be able to contribute to this PR any more. @zlonast: how can we help to get it through?

[zlonast]

@Mikolaj Hi! As the author, I think it works and we can be merge it 😄

But perhaps you want to pay attention to this:

1. The GCC issue: We want to explicitly pass the GCC path, but for versions < 9.4, it causes linking failures for reasons I haven't been able to isolate yet.
2. Lack of tests: I haven't implemented tests for the new flag options yet.

[ulysses4ever]

@zlonast thank you for your work here and for the summary in partiular. Quetsions:

The GCC issue: We want to explicitly pass the GCC path, but for versions < 9.4, it causes linking failures for reasons I haven't been able to isolate yet.

This looks rather serious. Can you elaborate with example scenarios? E.g. will I not be able to build any package with c-sources after this change if I need to use 9.4?

FTR, we are supposed to maintain ccompatibility with any GHC released last 5 years. GHC 9.4.8 was released 10 November 2023, so there's still 2.5 years of it staying in this support window.

Lack of tests: I haven't implemented tests for the new flag options yet.

Do you think you could add some?

[zlonast]

This must happen before this pull request is merged.

The GCC issue: We want to explicitly pass the GCC path, but for versions < 9.4, it causes linking failures for reasons I haven't been able to isolate yet.

issue #11712

pr fix #11713

Lack of tests: I haven't implemented tests for the new flag options yet.

tests #11711

[zlonast]

-optc-O2,-optcxx-O2,-opta-O2 into a separate pull request #11716

[zlonast]

I found the problem and left a link in this #10969 (comment).

I've only left two things in that pr.

1. the solution itself
2. code refactoring [significance: significant]

I suggest checking the compilation of the entire clc-stackage again before merging.