componentAvailableTargetStatus: impossible; cname=test

[phadej]

When a project uses local package tarballs, project planning fails.

Small reproducer:

take any package with tests, e.g. splitmix (I pick that because it depends only on bundled libraries): https://hackage.haskell.org/package/splitmix-0.1.0.4/splitmix-0.1.0.4.tar.gz

% ls
cabal.project  foo.cabal  splitmix-0.1.0.4.tar.gz

cabal.project

packages: .
packages: splitmix-0.1.0.4.tar.gz

foo.cabal

cabal-version: 3.6
name: foo
version: 0

library
  build-depends: base, splitmix

Then if you try to build the project enabling tests, it fails:

% ~/.ghcup/bin/cabal-3.9.0.0 build --enable-tests all
componentAvailableTargetStatus: impossible; cname=test:examples
CallStack (from HasCallStack):
  error, called at src/Distribution/Client/ProjectPlanning.hs:2698:17 in cabal-install-3.9.0.0-inplace:Distribution.Client.ProjectPlanning

The :test:examples component is a test-suite of splitmix. It shouldn't be required or built. This seems to be a regression since cabal-3.8.1.0.

[phadej]

The culprit commit is b547ead

The

diff --git a/cabal-install/src/Distribution/Client/ProjectPlanning.hs b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
index 4ec141037..4dbe4d74b 100644
--- a/cabal-install/src/Distribution/Client/ProjectPlanning.hs
+++ b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
@@ -2143,7 +2143,7 @@ isInLocal _extraPackages (SpecificSourcePackage pkg) = case srcpkgSource pkg of
     -- as these flags would apply to local packages, but the sdist would
     -- erroneously not get categorized as a local package, so the flags would be
     -- ignored and produce a package with an unchanged hash.
-    LocalTarballPackage  _ -> Just (packageId pkg)
+    -- LocalTarballPackage  _ -> Just (packageId pkg)
     -- TODO: the docs say ‘extra-packages’ is implemented in cabal project
     -- files.  We can fix that here by checking that the version range matches.
     --RemoteTarballPackage    _ -> _

Seems to fix the issue. I was right telling @Mikolaj in private that it's probably because someone consused local as in "on my machine" with local is in "project package". That is unfortunately illogical/ambiguous naming in cabal-install codebase. (THe LocalUnpackedPackage are the only "project packages", everything else is not).

[jneira]

another code path without regression test (and this seems to be an important one not too difficult to add) 😓

[gbaz]

A general instance of not having regression tests for cabal install commands almost entirely.

[bairyn]

Thanks for the report and simple scenario I could reproduce. I authored that commit along with the test that follows it. It seems to me it unmasks another error. Sorry that's given you issues.

To me it seems there are disagreeing perspectives on what ‘local’ means or what's better. I suppose the majority can decide, or policy can decide (e.g. 2 reviewers for this distribution), or in principle anyway forks could be made, if I'm thinking about this well.

Here's what I'm getting from the docs (source
https://cabal.readthedocs.io/en/stable/nix-local-build.html#developing-multiple-packages
and
https://cabal.readthedocs.io/en/stable/nix-local-build.html#how-it-works):

How it works
============

Local versus external packages
------------------------------

One of the primary innovations of Nix-style local builds is the
distinction between local packages, which users edit and recompile and
must be built per-project, versus external packages, which can be cached
across projects. To be more precise:

1. A **local package** is one that is listed explicitly in the
   ``packages``, ``optional-packages`` or ``extra-packages`` fields of a
   project. Packages in the former two fields will usually have their
   source code stored in a folder in your project, while ``extra-packages`` lists
   packages residing on Hackage that are treated as being local anyway.

Local packages, as well as the external packages (below) which depend on
them, are built **inplace**, meaning that they are always built
specifically for the project and are not installed globally. Inplace
packages are not cached and not given unique hashes, which makes them
suitable for packages which you want to edit and recompile.

2. An **external package** is any package which is not listed in the
   ``packages``, ``optional-packages`` and ``extra-packages`` fields.
   The source code for external packages is usually retrieved from Hackage.

[…]

This split motivates some of the UI choices for Nix-style local build
commands. For example, flags passed to ``cabal v2-build`` are only
applied to *local* packages, so that adding a flag to
``cabal v2-build`` doesn't necessitate a rebuild of *every* transitive
dependency in the global package store.

By my interpretation of the these docs, it seems it'd be required to me according to that approach.

Here's what I'm getting from the understanding that a package in a project is not local if it's an a tarball, but if it's unpacked then that package is local:

Then if you try to build the project enabling tests, it fails:

> % ~/.ghcup/bin/cabal-3.9.0.0 build --enable-tests all
> componentAvailableTargetStatus: impossible; cname=test:examples
> CallStack (from HasCallStack):
>   error, called at src/Distribution/Client/ProjectPlanning.hs:2698:17 in cabal-install-3.9.0.0-inplace:Distribution.Client.ProjectPlanning

The :test:examples component is a test-suite of splitmix. It shouldn't be required or built. This seems to be a regression since cabal-3.8.1.0.

You say:

The :test:examples component is a test-suite of splitmix. It shouldn't be required or built. This seems to be a regression since cabal-3.8.1.0.

and

Actually I think this change breaks the whole fix done #8623, the idea how to fix the issue there is wrong. local tarballs are not local packages, if you ask me, that's not negotiable.
[…]

If the spec that only calls unpacked intra-project packages local wins, then actually ‘--enable-tests’ should only apply presumably to building the referenced packages, although with ‘all’ specified to build all components of all packages, then I'd still expect ‘--enable-tests’ to apply to both the package referred to by ‘.’ and the package referred to by ‘splitmix-0.1.0.4.tar.gz’.

(Perhaps the interface could be updated to accommodate both interpretations, so that one build config flag affects all project-local packages, another affects globally all packages being built, another affects only project-local unpacked packages, and perhaps another affects only packages that meet the specified patterns.)

I think if it were put to a vote, I'd probably be in favor of the docs at least, but better (with the cost of it being more work) might be to support more use cases such as specifying the configuration for any package built, or to restrict the configuration to only specified packages.

(If it is further developed, while that would probably require work, D.C.ProjectPlanning.Types does seem to note an in-code task that could optionally be worked on:)

-- TODO: make per-component variants of these flags
elabVanillaLib           :: Bool,
elabSharedLib            :: Bool,
elabStaticLib            :: Bool,
elabDynExe               :: Bool,

Anyway, while I may disagree with that particular point, or at least on which perspective I prefer, it does look like there's a real error that's being brought up, which I can reproduce in your scenario.

The surrounding code looks like this:

-- it is not an optional stanza, so a testsuite or benchmark
Just stanza ->
  case (optStanzaLookup stanza (elabStanzasRequested elab), -- TODO
        optStanzaSetMember stanza (elabStanzasAvailable elab)) of
    _ | not withinPlan -> TargetNotLocal
    (Just False,   _)  -> TargetDisabledByUser
    (Nothing,  False)  -> TargetDisabledBySolver
    _ | not buildable  -> TargetNotBuildable
    (Just True, True)  -> TargetBuildable (elabUnitId elab, cname)
                                          TargetRequestedByDefault
    (Nothing,   True)  -> TargetBuildable (elabUnitId elab, cname)
                                          TargetNotRequestedByDefault
    (Just True, False) ->
      error $ "componentAvailableTargetStatus: impossible; cname=" ++ prettyShow cname

I took a look at it, and it looks like my older fix unmasks this ‘error’ branch, since the ‘TargetNotLocal’ branch was being hit too often - I think so, anyway. I wrote a quick draft of what seemed to be to be likely a good approach to fix this issue and pushed it to my ‘wip-target-not-available’ branch with the following diff:

From 7037409ceca8478b1020ae780e27f56ce269b95c Mon Sep 17 00:00:00 2001
From: Byron Johnson <byron@byronjohnson.net>
Date: Sat, 11 Feb 2023 21:25:29 -0700
Subject: [PATCH] WIP: fix #8719.

On my system, all tests passed 100%, and this also seemed to fix the error in
the sceanior mentioned, but this needs to be completed.  (I guess also double
check the surrounding code to make sure it's a good approach.)
---
 cabal-install/src/Distribution/Client/CmdErrorMessages.hs   | 5 +++++
 .../src/Distribution/Client/ProjectOrchestration.hs         | 3 +++
 cabal-install/src/Distribution/Client/ProjectPlanning.hs    | 6 +++++-
 cabal-install/src/Distribution/Client/TargetProblem.hs      | 3 +++
 4 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/cabal-install/src/Distribution/Client/CmdErrorMessages.hs b/cabal-install/src/Distribution/Client/CmdErrorMessages.hs
index cf41fc4eb..4e07d1df3 100644
--- a/cabal-install/src/Distribution/Client/CmdErrorMessages.hs
+++ b/cabal-install/src/Distribution/Client/CmdErrorMessages.hs
@@ -256,6 +256,9 @@ renderTargetProblem verb _ (TargetComponentNotBuildable pkgid cname _) =
  ++ "edit the .cabal file to declare it as buildable and fix any resulting "
  ++ "build problems."
 
+-- TODO
+renderTargetProblem _verb _ (TargetComponentNotAvailable _pkgid _cname _) = "TODO: DEBUG89: renderTargetProblem message for Not Available."
+
 renderTargetProblem verb _ (TargetOptionalStanzaDisabledByUser _ cname _) =
     "Cannot " ++ verb ++ " the " ++ showComponentName cname ++ " because "
  ++ "building " ++ compkinds ++ " has been explicitly disabled in the "
@@ -350,6 +353,7 @@ renderTargetProblemNoneEnabled verb targetSelector targets =
               | AvailableTarget {availableTargetComponentName} <- targets' ]
          ++ plural (listPlural targets') " is " " are all "
          ++ "marked as 'buildable: False'"
+        (TargetNotAvailable, _) -> "TODO NOT AVAILABLE DEBUG88"
         (TargetNotLocal, _) ->
             renderListCommaAnd
               [ "the " ++ showComponentName availableTargetComponentName
@@ -373,6 +377,7 @@ renderTargetProblemNoneEnabled verb targetSelector targets =
       , case availableTargetStatus t of
           TargetNotBuildable -> Nothing
           TargetNotLocal     -> Nothing
+          TargetNotAvailable -> Nothing  -- TODO
           _ -> optionalStanza (availableTargetComponentName t)
       )
 
diff --git a/cabal-install/src/Distribution/Client/ProjectOrchestration.hs b/cabal-install/src/Distribution/Client/ProjectOrchestration.hs
index da67b8a3e..6daa01639 100644
--- a/cabal-install/src/Distribution/Client/ProjectOrchestration.hs
+++ b/cabal-install/src/Distribution/Client/ProjectOrchestration.hs
@@ -826,6 +826,9 @@ selectComponentTargetBasic subtarget
       TargetNotBuildable ->
         Left (TargetComponentNotBuildable pkgid cname subtarget)
 
+      TargetNotAvailable ->
+        Left (TargetComponentNotAvailable pkgid cname subtarget)  -- TODO
+
       TargetBuildable targetKey _ ->
         Right targetKey
 
diff --git a/cabal-install/src/Distribution/Client/ProjectPlanning.hs b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
index 20d20795e..1ad3eb46c 100644
--- a/cabal-install/src/Distribution/Client/ProjectPlanning.hs
+++ b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
@@ -2553,6 +2553,7 @@ data AvailableTargetStatus k =
      | TargetNotBuildable     -- ^ When the component has @buildable: False@
      | TargetNotLocal         -- ^ When the component is non-core in a non-local package
      | TargetBuildable k TargetRequested -- ^ The target can or should be built
+     | TargetNotAvailable     -- ^ TODO
   deriving (Eq, Ord, Show, Functor)
 
 -- | This tells us whether a target ought to be built by default, or only if
@@ -2699,7 +2700,10 @@ availableSourceTargets elab =
               (Nothing,   True)  -> TargetBuildable (elabUnitId elab, cname)
                                                     TargetNotRequestedByDefault
               (Just True, False) ->
-                error $ "componentAvailableTargetStatus: impossible; cname=" ++ prettyShow cname
+                -- TODO: review & become more familiar with context to make
+                -- sure this makes sense.
+                --error $ "componentAvailableTargetStatus: impossible; cname=" ++ prettyShow cname
+                TargetNotAvailable
       where
         cname      = componentName component
         buildable  = PD.buildable (componentBuildInfo component)
diff --git a/cabal-install/src/Distribution/Client/TargetProblem.hs b/cabal-install/src/Distribution/Client/TargetProblem.hs
index 14004d50a..01de28541 100644
--- a/cabal-install/src/Distribution/Client/TargetProblem.hs
+++ b/cabal-install/src/Distribution/Client/TargetProblem.hs
@@ -21,6 +21,9 @@ data TargetProblem a
     | TargetComponentNotProjectLocal
       PackageId ComponentName SubComponentTarget
 
+    | TargetComponentNotAvailable  -- TODO
+      PackageId ComponentName SubComponentTarget
+
     | TargetComponentNotBuildable
       PackageId ComponentName SubComponentTarget
 
-- 
2.39.0

I'd probably like to double check the code around it to verify it seems to make sense, as that particular part of the code is one I haven't worked through as much as some of the other parts, and the various TODO messages and details would need to be filled/worked out.

But it did seem to fix the error in your scenario.

If nobody beats me to it, I'll likely have time available to set aside for this next weekend (last weekend unfortunately other things prevented me from being able to contribute to Cabal that weekend).

Now if you're in a rush to get the next release out before this is done and would rather revert the fix at least in the meantime, trading one error for another, I think that's fine and don't want to get in the way of slowing it down. But given that it's OSS and I already submitted the contributions, I'm not sure it'd much matter anyway. Or if you want me to submit a GitHub approval to get it reverted, I guess I can do that too. I could just be one part of the majority, outside my own fork.

[bairyn]

(And one more note: IIRC, WRT the original fix, ‘cabal-install v2-install’ looked like it was producing its own sdist from the unpacked package and the build flags were getting dropped, but I might need to double check that.) (Also tagging people likely to be interested in this update: @Mikolaj @gbaz (I didn't notice this discussion on this merged PR until I was pinged, so maybe I could also practice utilizing this GitHub feature better.))