Moving location where library build files are generated

[ezyang]

Summary by @ezyang. We considered the following options for restructuring dist/build such that component build directories are not a subdirectory of the library build directory:

1. Move library directory to dist/build/pkgname. Problem: This will conflict with an executable that is named the same as the package.
2. Move non-library components to dist/subbuild/compname. Problem: This moves where executables are, and many people (including this author) are used to accessing the executable at dist/build/exename/exename. One way to patch over this is to copy the executables back to the old locations.
3. Move the library directory to dist/build/lib-pkgname. Problem: This will still conflict with a component named lib-pkgname.
4. Qualify all components with their type, e.g., dist/build/exe-exename. Problem: This moves where the executables are.
5. Move the library directory to dist/libbuild. No major problems, although @23Skidoo would prefer all build products be in dist/build.

We are going to go with proposal (5).

---

Hey guys. I want to propose a BC-breaking change to the dist directory layout. I'll first explain what the change is, and then why I need it. I'd like to know if people are OK with it. It WILL break code which needs to know where Cabal puts build products. CC @mgsloan, @23Skidoo , @dcoutts

Cabal-1.24 behavior. Suppose we are building a Haskell module M.hs. If this module is in the public library, the interface file is placed in dist/build/M.hi. If the module is in any other component (e.g., a test suite) named comp, the interface file is placed in dist/build/comp/M.hi)

Proposed new behavior. I propose that libraries also get their own subdirectory: dist/build/pkgname; thus if we build M.hs in a library named foobar it will be placed in dist/build/foobar/M.hi. Obviously, this will break any scripts which rely on the interface files/object files having been placed in the old location.

Motivation. Cabal supports pre-processors which may generate extra object files that need to be linked into the final libraries/executables we build. This is tested by the PreProcessExtraSources cabal:package-test. For example, if we build Foo.hsc, this results in a Foo_hsc.o in our build directory.

If a preprocessor generates this extra object file, how does Cabal know to link it in? The "obvious" design is that when you run the preprocessor, it tells you what extra files need to be compiled and you go ahead and compile them. But this is not what Cabal does. Instead, what it does is simply look in the build directory for any files which look like ones that the preprocessor generated, and make sure to link them in. (ian-ross@e38cb0c)

Now here is the problem with Cabal-1.24's behavior: the build directories for the test suites/benchmarks/etc are all INSIDE the library's build directory. So clearly the search code will pick up the extra build products: disaster! And so the most obvious fix (without completely changing the preprocessor interface) is to move the directory out of the way.

The bug got uncovered by #3542 although I am not really sure why it didn't manifest earlier.

Alternatives. I think there are two plausible alternatives which avoid moving the location of library files:

1. Move all non-library components to another directory, e.g., "subbuild" in dist. Still a BC break, because the binary for executables is no longer where you expect it to be.
2. Keep the current layout, and filter out subdirectories which correspond to other components in the Cabal file. I am morally opposed to this option, since it enshrines a simply incorrect design decision. Look at this Cabal file:

name:                xyz
version:             0.1.0.0
license:             BSD3
author:              Edward Z. Yang
maintainer:          ezyang@cs.stanford.edu
extra-source-files:  ChangeLog.md
cabal-version:       >=1.10

library
  exposed-modules:     Doom.Foo
  build-depends:       base
  default-language:    Haskell2010

test-suite Doom
    main-is: Foo.hs
    type: exitcode-stdio-1.0
    build-depends: base

These two Foos clobber each other! Ouch!

[phadej]

Could this be done for new-build only, and how it relates to it?

[23Skidoo]

I propose that libraries also get their own subdirectory: dist/build/pkgname; thus if we build M.hs in a library named foobar it will be placed in dist/build/foobar/M.hi.

Will this still work when there's an executable with the same name as library (example)? Otherwise +1.

[ezyang]

@phadej : No, it affects both new-build and build; it's a modification to how Setup scripts work.

@23Skidoo: Oh right. You'll need to disambiguate the public library somehow. The subbuild alternative plan would do this (at the cost of moving where all the executables are.) Maybe we should do that then.

[ezyang]

OK, I came up with another "more BC" scheme:

1. The public library continues to be built into dist/build
2. Other components are built into dist/subbuild/$component_name
3. However, all components which produce executables have the executable copied into dist/build/$component_name/$exe_name

So, no library files move, and the executables are where you expect them.

[ezyang]

The bug that motivated this is present in master; however it doesn't show up on a fresh build because the benchmarks/test suites get built after the library, avoiding the library linking step from picking up the duplicate objects.

[phadej]

btw, just today I run into situations where two executables in the same package use the same cabal_macros.h, so MIN_VERSION_package flags from other one leaks into other one.

EDIT in particularly I cannot have a source file with "clever" #ifdef MIN_VERSION_foo definitions

So the fix for this is very welcome!

[ezyang]

The cabal_macros.h but is different and already fixed in master, see #1893 and the fixing commit is 90e908b

[23Skidoo]

I'd prefer not to add a subbuild directory, IMO the build/subbuild distinction is confusing. Can we put library files into e.g. build/lib-foo and leave everything else as before? We'll have to prohibit components that have a name starting with lib-, but I don't think it's a big problem in practice.