Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

componentAvailableTargetStatus: impossible; cname=test #8719

Closed
phadej opened this issue Feb 1, 2023 · 11 comments
Closed

componentAvailableTargetStatus: impossible; cname=test #8719

phadej opened this issue Feb 1, 2023 · 11 comments

Comments

@phadej
Copy link
Collaborator

phadej commented Feb 1, 2023

When a project uses local package tarballs, project planning fails.

Small reproducer:

take any package with tests, e.g. splitmix (I pick that because it depends only on bundled libraries): https://hackage.haskell.org/package/splitmix-0.1.0.4/splitmix-0.1.0.4.tar.gz

% ls
cabal.project  foo.cabal  splitmix-0.1.0.4.tar.gz

cabal.project

packages: .
packages: splitmix-0.1.0.4.tar.gz

foo.cabal

cabal-version: 3.6
name: foo
version: 0

library
  build-depends: base, splitmix

Then if you try to build the project enabling tests, it fails:

% ~/.ghcup/bin/cabal-3.9.0.0 build --enable-tests all
componentAvailableTargetStatus: impossible; cname=test:examples
CallStack (from HasCallStack):
  error, called at src/Distribution/Client/ProjectPlanning.hs:2698:17 in cabal-install-3.9.0.0-inplace:Distribution.Client.ProjectPlanning

The :test:examples component is a test-suite of splitmix. It shouldn't be required or built. This seems to be a regression since cabal-3.8.1.0.

@phadej
Copy link
Collaborator Author

phadej commented Feb 1, 2023

The culprit commit is b547ead

The

diff --git a/cabal-install/src/Distribution/Client/ProjectPlanning.hs b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
index 4ec141037..4dbe4d74b 100644
--- a/cabal-install/src/Distribution/Client/ProjectPlanning.hs
+++ b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
@@ -2143,7 +2143,7 @@ isInLocal _extraPackages (SpecificSourcePackage pkg) = case srcpkgSource pkg of
     -- as these flags would apply to local packages, but the sdist would
     -- erroneously not get categorized as a local package, so the flags would be
     -- ignored and produce a package with an unchanged hash.
-    LocalTarballPackage  _ -> Just (packageId pkg)
+    -- LocalTarballPackage  _ -> Just (packageId pkg)
     -- TODO: the docs say ‘extra-packages’ is implemented in cabal project
     -- files.  We can fix that here by checking that the version range matches.
     --RemoteTarballPackage    _ -> _

Seems to fix the issue. I was right telling @Mikolaj in private that it's probably because someone consused local as in "on my machine" with local is in "project package". That is unfortunately illogical/ambiguous naming in cabal-install codebase. (THe LocalUnpackedPackage are the only "project packages", everything else is not).

@jneira
Copy link
Member

jneira commented Feb 3, 2023

another code path without regression test (and this seems to be an important one not too difficult to add) 😓

@gbaz
Copy link
Collaborator

gbaz commented Feb 3, 2023

A general instance of not having regression tests for cabal install commands almost entirely.

bairyn added a commit to bairyn/cabal that referenced this issue Feb 12, 2023
On my system, all tests passed 100%, and this also seemed to fix the error in
the sceanior mentioned, but this needs to be completed.  (I guess also double
check the surrounding code to make sure it's a good approach.)
@bairyn
Copy link
Collaborator

bairyn commented Feb 12, 2023

Thanks for the report and simple scenario I could reproduce. I authored that commit along with the test that follows it. It seems to me it unmasks another error. Sorry that's given you issues.

To me it seems there are disagreeing perspectives on what ‘local’ means or what's better. I suppose the majority can decide, or policy can decide (e.g. 2 reviewers for this distribution), or in principle anyway forks could be made, if I'm thinking about this well.

Here's what I'm getting from the docs (source
https://cabal.readthedocs.io/en/stable/nix-local-build.html#developing-multiple-packages
and
https://cabal.readthedocs.io/en/stable/nix-local-build.html#how-it-works):

How it works
============

Local versus external packages
------------------------------

One of the primary innovations of Nix-style local builds is the
distinction between local packages, which users edit and recompile and
must be built per-project, versus external packages, which can be cached
across projects. To be more precise:

1. A **local package** is one that is listed explicitly in the
   ``packages``, ``optional-packages`` or ``extra-packages`` fields of a
   project. Packages in the former two fields will usually have their
   source code stored in a folder in your project, while ``extra-packages`` lists
   packages residing on Hackage that are treated as being local anyway.

Local packages, as well as the external packages (below) which depend on
them, are built **inplace**, meaning that they are always built
specifically for the project and are not installed globally. Inplace
packages are not cached and not given unique hashes, which makes them
suitable for packages which you want to edit and recompile.

2. An **external package** is any package which is not listed in the
   ``packages``, ``optional-packages`` and ``extra-packages`` fields.
   The source code for external packages is usually retrieved from Hackage.

[…]

This split motivates some of the UI choices for Nix-style local build
commands. For example, flags passed to ``cabal v2-build`` are only
applied to *local* packages, so that adding a flag to
``cabal v2-build`` doesn't necessitate a rebuild of *every* transitive
dependency in the global package store.


By my interpretation of the these docs, it seems it'd be required to me according to that approach.

Here's what I'm getting from the understanding that a package in a project is not local if it's an a tarball, but if it's unpacked then that package is local:

Then if you try to build the project enabling tests, it fails:

> % ~/.ghcup/bin/cabal-3.9.0.0 build --enable-tests all
> componentAvailableTargetStatus: impossible; cname=test:examples
> CallStack (from HasCallStack):
>   error, called at src/Distribution/Client/ProjectPlanning.hs:2698:17 in cabal-install-3.9.0.0-inplace:Distribution.Client.ProjectPlanning

The :test:examples component is a test-suite of splitmix. It shouldn't be required or built. This seems to be a regression since cabal-3.8.1.0.

You say:

The :test:examples component is a test-suite of splitmix. It shouldn't be required or built. This seems to be a regression since cabal-3.8.1.0.

and

Actually I think this change breaks the whole fix done #8623, the idea how to fix the issue there is wrong. local tarballs are not local packages, if you ask me, that's not negotiable.
[…]

If the spec that only calls unpacked intra-project packages local wins, then actually ‘--enable-tests’ should only apply presumably to building the referenced packages, although with ‘all’ specified to build all components of all packages, then I'd still expect ‘--enable-tests’ to apply to both the package referred to by ‘.’ and the package referred to by ‘splitmix-0.1.0.4.tar.gz’.

(Perhaps the interface could be updated to accommodate both interpretations, so that one build config flag affects all project-local packages, another affects globally all packages being built, another affects only project-local unpacked packages, and perhaps another affects only packages that meet the specified patterns.)

I think if it were put to a vote, I'd probably be in favor of the docs at least, but better (with the cost of it being more work) might be to support more use cases such as specifying the configuration for any package built, or to restrict the configuration to only specified packages.

(If it is further developed, while that would probably require work, D.C.ProjectPlanning.Types does seem to note an in-code task that could optionally be worked on:)

-- TODO: make per-component variants of these flags
elabVanillaLib           :: Bool,
elabSharedLib            :: Bool,
elabStaticLib            :: Bool,
elabDynExe               :: Bool,

Anyway, while I may disagree with that particular point, or at least on which perspective I prefer, it does look like there's a real error that's being brought up, which I can reproduce in your scenario.

The surrounding code looks like this:

-- it is not an optional stanza, so a testsuite or benchmark
Just stanza ->
  case (optStanzaLookup stanza (elabStanzasRequested elab), -- TODO
        optStanzaSetMember stanza (elabStanzasAvailable elab)) of
    _ | not withinPlan -> TargetNotLocal
    (Just False,   _)  -> TargetDisabledByUser
    (Nothing,  False)  -> TargetDisabledBySolver
    _ | not buildable  -> TargetNotBuildable
    (Just True, True)  -> TargetBuildable (elabUnitId elab, cname)
                                          TargetRequestedByDefault
    (Nothing,   True)  -> TargetBuildable (elabUnitId elab, cname)
                                          TargetNotRequestedByDefault
    (Just True, False) ->
      error $ "componentAvailableTargetStatus: impossible; cname=" ++ prettyShow cname

I took a look at it, and it looks like my older fix unmasks this ‘error’ branch, since the ‘TargetNotLocal’ branch was being hit too often - I think so, anyway. I wrote a quick draft of what seemed to be to be likely a good approach to fix this issue and pushed it to my ‘wip-target-not-available’ branch with the following diff:

From 7037409ceca8478b1020ae780e27f56ce269b95c Mon Sep 17 00:00:00 2001
From: Byron Johnson <[email protected]>
Date: Sat, 11 Feb 2023 21:25:29 -0700
Subject: [PATCH] WIP: fix #8719.

On my system, all tests passed 100%, and this also seemed to fix the error in
the sceanior mentioned, but this needs to be completed.  (I guess also double
check the surrounding code to make sure it's a good approach.)
---
 cabal-install/src/Distribution/Client/CmdErrorMessages.hs   | 5 +++++
 .../src/Distribution/Client/ProjectOrchestration.hs         | 3 +++
 cabal-install/src/Distribution/Client/ProjectPlanning.hs    | 6 +++++-
 cabal-install/src/Distribution/Client/TargetProblem.hs      | 3 +++
 4 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/cabal-install/src/Distribution/Client/CmdErrorMessages.hs b/cabal-install/src/Distribution/Client/CmdErrorMessages.hs
index cf41fc4eb..4e07d1df3 100644
--- a/cabal-install/src/Distribution/Client/CmdErrorMessages.hs
+++ b/cabal-install/src/Distribution/Client/CmdErrorMessages.hs
@@ -256,6 +256,9 @@ renderTargetProblem verb _ (TargetComponentNotBuildable pkgid cname _) =
  ++ "edit the .cabal file to declare it as buildable and fix any resulting "
  ++ "build problems."
 
+-- TODO
+renderTargetProblem _verb _ (TargetComponentNotAvailable _pkgid _cname _) = "TODO: DEBUG89: renderTargetProblem message for Not Available."
+
 renderTargetProblem verb _ (TargetOptionalStanzaDisabledByUser _ cname _) =
     "Cannot " ++ verb ++ " the " ++ showComponentName cname ++ " because "
  ++ "building " ++ compkinds ++ " has been explicitly disabled in the "
@@ -350,6 +353,7 @@ renderTargetProblemNoneEnabled verb targetSelector targets =
               | AvailableTarget {availableTargetComponentName} <- targets' ]
          ++ plural (listPlural targets') " is " " are all "
          ++ "marked as 'buildable: False'"
+        (TargetNotAvailable, _) -> "TODO NOT AVAILABLE DEBUG88"
         (TargetNotLocal, _) ->
             renderListCommaAnd
               [ "the " ++ showComponentName availableTargetComponentName
@@ -373,6 +377,7 @@ renderTargetProblemNoneEnabled verb targetSelector targets =
       , case availableTargetStatus t of
           TargetNotBuildable -> Nothing
           TargetNotLocal     -> Nothing
+          TargetNotAvailable -> Nothing  -- TODO
           _ -> optionalStanza (availableTargetComponentName t)
       )
 
diff --git a/cabal-install/src/Distribution/Client/ProjectOrchestration.hs b/cabal-install/src/Distribution/Client/ProjectOrchestration.hs
index da67b8a3e..6daa01639 100644
--- a/cabal-install/src/Distribution/Client/ProjectOrchestration.hs
+++ b/cabal-install/src/Distribution/Client/ProjectOrchestration.hs
@@ -826,6 +826,9 @@ selectComponentTargetBasic subtarget
       TargetNotBuildable ->
         Left (TargetComponentNotBuildable pkgid cname subtarget)
 
+      TargetNotAvailable ->
+        Left (TargetComponentNotAvailable pkgid cname subtarget)  -- TODO
+
       TargetBuildable targetKey _ ->
         Right targetKey
 
diff --git a/cabal-install/src/Distribution/Client/ProjectPlanning.hs b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
index 20d20795e..1ad3eb46c 100644
--- a/cabal-install/src/Distribution/Client/ProjectPlanning.hs
+++ b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
@@ -2553,6 +2553,7 @@ data AvailableTargetStatus k =
      | TargetNotBuildable     -- ^ When the component has @buildable: False@
      | TargetNotLocal         -- ^ When the component is non-core in a non-local package
      | TargetBuildable k TargetRequested -- ^ The target can or should be built
+     | TargetNotAvailable     -- ^ TODO
   deriving (Eq, Ord, Show, Functor)
 
 -- | This tells us whether a target ought to be built by default, or only if
@@ -2699,7 +2700,10 @@ availableSourceTargets elab =
               (Nothing,   True)  -> TargetBuildable (elabUnitId elab, cname)
                                                     TargetNotRequestedByDefault
               (Just True, False) ->
-                error $ "componentAvailableTargetStatus: impossible; cname=" ++ prettyShow cname
+                -- TODO: review & become more familiar with context to make
+                -- sure this makes sense.
+                --error $ "componentAvailableTargetStatus: impossible; cname=" ++ prettyShow cname
+                TargetNotAvailable
       where
         cname      = componentName component
         buildable  = PD.buildable (componentBuildInfo component)
diff --git a/cabal-install/src/Distribution/Client/TargetProblem.hs b/cabal-install/src/Distribution/Client/TargetProblem.hs
index 14004d50a..01de28541 100644
--- a/cabal-install/src/Distribution/Client/TargetProblem.hs
+++ b/cabal-install/src/Distribution/Client/TargetProblem.hs
@@ -21,6 +21,9 @@ data TargetProblem a
     | TargetComponentNotProjectLocal
       PackageId ComponentName SubComponentTarget
 
+    | TargetComponentNotAvailable  -- TODO
+      PackageId ComponentName SubComponentTarget
+
     | TargetComponentNotBuildable
       PackageId ComponentName SubComponentTarget
 
-- 
2.39.0

I'd probably like to double check the code around it to verify it seems to make sense, as that particular part of the code is one I haven't worked through as much as some of the other parts, and the various TODO messages and details would need to be filled/worked out.

But it did seem to fix the error in your scenario.

If nobody beats me to it, I'll likely have time available to set aside for this next weekend (last weekend unfortunately other things prevented me from being able to contribute to Cabal that weekend).

Now if you're in a rush to get the next release out before this is done and would rather revert the fix at least in the meantime, trading one error for another, I think that's fine and don't want to get in the way of slowing it down. But given that it's OSS and I already submitted the contributions, I'm not sure it'd much matter anyway. Or if you want me to submit a GitHub approval to get it reverted, I guess I can do that too. I could just be one part of the majority, outside my own fork.

@bairyn
Copy link
Collaborator

bairyn commented Feb 12, 2023

(And one more note: IIRC, WRT the original fix, ‘cabal-install v2-install’ looked like it was producing its own sdist from the unpacked package and the build flags were getting dropped, but I might need to double check that.) (Also tagging people likely to be interested in this update: @Mikolaj @gbaz (I didn't notice this discussion on this merged PR until I was pinged, so maybe I could also practice utilizing this GitHub feature better.))

@Mikolaj
Copy link
Member

Mikolaj commented Feb 13, 2023

@bairyn: thank you very much for having a look again. What do you think about reverting half of the PR in #8744 (not reverting the test, though) so that you have ample time to revisit the changes, split them up and we can review them slowly one by one and with better understanding (I apparently lacked sufficient understanding when reviewing last time)? I think there is no point to rush and a revert would take the pressure of from us.

@bairyn
Copy link
Collaborator

bairyn commented Feb 14, 2023

Okay.

@Mikolaj
Copy link
Member

Mikolaj commented Feb 14, 2023

OK, we have a clean slate, we can start adding things back. @bairyn: please feel free to re-add stuff and please be mindful some us (e.g., me) require the innovation to be served in very small bites to review them effectively. It would be great if we could also add tests to each of these features, because this old PR was a very important one, touching/improving very important areas, but the high value comes with high risk and tests mitigate that.

@gbaz: I remember you were especially keen to bring one of the features from the old PR back, perhaps even into 3.10, if we manage to in time. Which one was that?

@Mikolaj
Copy link
Member

Mikolaj commented Feb 14, 2023

Could you kindly check if the bug if perhaps gone in this version with some fixes: https://gitlab.haskell.org/haskell/cabal/-/pipelines/63307

The commandline to get the Linux version via ghcup is
ghcup --no-cache install cabal -u 'https://gitlab.haskell.org/haskell/cabal/-/jobs/1376184/artifacts/raw/out/cabal-install-3.9-x86_64-linux-alpine.tar.xz' internal-second-prerelease-of-3.10

@gbaz
Copy link
Collaborator

gbaz commented Feb 15, 2023

should be resolved by #8744

@gbaz gbaz closed this as completed Feb 15, 2023
@gbaz
Copy link
Collaborator

gbaz commented Feb 15, 2023

@gbaz: I remember you were especially keen to bring one of the features from the old PR back, perhaps even into 3.10, if we manage to in time. Which one was that?

As I wrote on the other ticket, the important thing from the reverted pr was "teaching cabal to use the same "way" (i.e. dynamic or static) to build the setup script as it uses to build the package itself".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants