Cabal install fails when ** wildcards are used

[mrkkrp]

See NixOS/nixpkgs#64173 (comment)

[quasicomputational]

I can't replicate this locally with just v2-install (i.e., no Nix). v1-install is chugging away but it'll be a minute because it's installing about five million things. (And, it worked, so I can't reproduce it at all, it seems.)

-v3 output may be helpful, if someone who can replicate the failure (maybe @cdepillabout?) could do that.

[cdepillabout]

@quasicomputational The full cabal v2-install -v3output was quite verbose, so I put it in a gist:

https://gist.github.com/cdepillabout/9a2ca42fcf78fa9462b63ea3c4a8f533#file-cabal-install-output-with-error-txt

This is on NixOS, getting GHC and cabal from Nix (using the shell.nix in ormolu's repo).

This is using the following cabal:

$ cabal --version
cabal-install version 2.4.1.0
compiled using version 2.4.1.0 of the Cabal library 

[quasicomputational]

Thanks!

Here's what look like the critical lines:

Installing ./data/printer/ssig-single.hs to
/home/illabout/.cabal/store/ghc-8.6.4/incoming/new-15111/home/illabout/.cabal/store/ghc-8.6.4/ormolu-0.0.1.0-92a9154ca931fd51a93dff32cdbea6dc0ddb6dbdc5d057c74b04741d826ca4f7/share/data/printer/ssig-single.hs
/home/illabout/.cabal/store/ghc-8.6.4/incoming/new-15111/home/illabout/.cabal/store/ghc-8.6.4/ormolu-0.0.1.0-92a9154ca931fd51a93dff32cdbea6dc0ddb6dbdc5d057c74b04741d826ca4f7/share/data/printer: copyFile: does not exist (No such file or directory)
CallStack (from HasCallStack):
die', called at ./Distribution/Client/ProjectOrchestration.hs:1002:55 in main:Distribution.Client.ProjectOrchestration

To me, that looks like copyFile is trying to act on a directory and failing, which sounds sort of plausible if somehow the globbing code is broken. That line is very likely being logged from installOrdinaryFile, which has a few call sites, but a reasonable guess is that Distribution.Simple.Install is one of the culprits, and in particular a function called installDataFiles stands out as a candidate.

Unfortunately, the code there looks reasonable to me and just eyeballing it doesn't reveal an obvious bug.

I don't understand why this would only be occurring in a Nix environment - when I try it locally, I don't get that second Installing line only naming the directory, but only ones naming files.

Just to confirm where the bug is, could you run this script (with cabal new-run test.hs) in the ormolu directory and put the output in a gist, please?

#!/usr/bin/env cabal
{- cabal:
build-depends: base ^>= 4.12
             , Cabal ^>= 2.4
-}
import Data.Foldable
import Distribution.Simple.Glob
import Distribution.Verbosity
import Distribution.Version

main =
  matchDirFileGlob normal (mkVersion [2,4]) "." "data/**/*.hs" >>= traverse_ putStrLn

[cdepillabout]

@quasicomputational Here's the output of cabal new-run test.hs in the ormolu directory:

https://gist.github.com/cdepillabout/9a2ca42fcf78fa9462b63ea3c4a8f533#file-output-of-test-hs-txt

It looks like it is listing all the files correctly, so I'm not sure what could be going on here :-\

[cdepillabout]

@quasicomputational

Here's the function you linked to:

cabal/Cabal/Distribution/Simple/Install.hs

Lines 224 to 236 in ed3ae13

	installDataFiles :: Verbosity -> PackageDescription -> FilePath -> IO ()
	installDataFiles verbosity pkg_descr destDataDir =
	flip traverse_ (dataFiles pkg_descr) $ \ file -> do
	let srcDataDirRaw = dataDir pkg_descr
	srcDataDir = if null srcDataDirRaw
	then "."
	else srcDataDirRaw
	files <- matchDirFileGlob verbosity (specVersion pkg_descr) srcDataDir file
	let dir = takeDirectory file
	createDirectoryIfMissingVerbose verbosity True (destDataDir </> dir)
	sequence_ [ installOrdinaryFile verbosity (srcDataDir </> file')
	(destDataDir </> file')
	| file' <- files ]

Particularly these lines:

cabal/Cabal/Distribution/Simple/Install.hs

Lines 231 to 233 in ed3ae13

	files <- matchDirFileGlob verbosity (specVersion pkg_descr) srcDataDir file
	let dir = takeDirectory file
	createDirectoryIfMissingVerbose verbosity True (destDataDir </> dir)

And here is the relevant line from your test.hs file:

main =
  matchDirFileGlob normal (mkVersion [2,4]) "." "data/**/*.hs" >>= traverse_ putStrLn

In Cabal.Distribution.Simple.Install, if file is just the filepath with globs (in this case data/**/*.hs, then maybe takeDirectory file will return something like data/**? Then when createDirectoryIfMissing is called, it tries to literally create data/**?

[quasicomputational]

So, that makes sense why this is failing. And I just worked out why I couldn't replicate it: I had the wrong branch cheked out. Whoops. Thanks for the remote debugging support!

I'll put together a test and a patch for this.

(Also, this bug would've been caught if globs were a newtype instead of FilePath. I had that idea on the backburner for a while, but seeing it actually cause problems definitely gives it a bump.)

[mrkkrp]

@quasicomputational So when we'll be able to use the fix?

[quasicomputational]

Hopefully soon, when Cabal 3.0 comes out with GHC 8.8! But one snag is that users will still have 2.4 installs for a while, so, yeah.

[23Skidoo]

I can make one final lib:Cabal 2.4 point release if this helps.

[mrkkrp]

If that will allow users to build cabal install without the bug, that would be awesome.

[mrkkrp]

@23Skidoo Is that release coming?

[phadej]

From what I know, upgrading a library bundled with GHC is a PITA. Yet, I heard that Nix is good at pulling stuff from git repositories. Could someone try first, and confirm that a release would actually help?

[mrkkrp]

I'll try today or tomorrow.

[23Skidoo]

cabal-install certainly can use a version of lib:Cabal different from the one that comes with GHC to compile custom Setups. But I have no idea about Nix.

[mrkkrp]

I tried this today but apparently ghc-paths do not like this version of Cabal:

Replace Cabal file with edited version from http://hackage.haskell.org/package/ghc-paths-0.1.0.9/revision/4.cabal.
compileBuildDriverPhase
setupCompileFlags: -package-db=/build/setup-package.conf.d -j4 -threaded
[1 of 1] Compiling Main             ( Setup.hs, /build/Main.o )

Setup.hs:29:18: error:
    Variable not in scope:
      rawSystemProgramStdoutConf
        :: Distribution.Verbosity.Verbosity
           -> Program -> ProgramDb -> [[Char]] -> IO [Char]
   |
29 |       libdir_ <- rawSystemProgramStdoutConf (fromFlag (configVerbosity flags))
   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
builder for '/nix/store/4gs6n5vxx1dlj8bql53qyhl32k908vb3-ghc-paths-0.1.0.9.drv' failed with exit code 1
cannot build derivation '/nix/store/m3jqba979p2v16cpjcanb1pqq7gglv25-ghc-8.6.4-with-packages.drv': 1 dependencies couldn't be built

I don't have time to make the whole tree build against this Cabal.

Here is the code:

• https://github.com/tweag/ormolu/pull/149/files

[andreasabel]

Issue summary:

• Directory globs in data-files, e.g., data-files: foo/**/bar.txt are buggy in Cabal-2.4.
• Work correctly since Cabal-3.0.