Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal libraries break (?) package.conf.inplace #5857

Closed
michaelpj opened this issue Jan 23, 2019 · 11 comments
Closed

Internal libraries break (?) package.conf.inplace #5857

michaelpj opened this issue Jan 23, 2019 · 11 comments
Labels
re: internal library Concerning internal libraries in packages type: bug

Comments

@michaelpj
Copy link
Collaborator

Observed using Cabal 2.4.1.

Reproduction case here: https://github.com/michaelpj/cabal-bug-package-db

If I add even an empty internal library to my package, the package db in dist/package.conf.inplace seems to no longer work. ghc-pkg list shows the packages as there and exposed, but trying to use them with ghc -package-db does not work.

michaelpj added a commit to michaelpj/plutus that referenced this issue Jan 23, 2019
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
michaelpj added a commit to michaelpj/plutus that referenced this issue Jan 23, 2019
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer. We can then organize
things a bit more neatly inside.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
michaelpj added a commit to michaelpj/plutus that referenced this issue Jan 23, 2019
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer. We can then organize
things a bit more neatly inside.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
michaelpj added a commit to michaelpj/plutus that referenced this issue Jan 23, 2019
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer. We can then organize
things a bit more neatly inside.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
michaelpj added a commit to michaelpj/plutus that referenced this issue Jan 23, 2019
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer. We can then organize
things a bit more neatly inside.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
michaelpj added a commit to michaelpj/plutus that referenced this issue Jan 23, 2019
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer. We can then organize
things a bit more neatly inside.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
@hvr
Copy link
Member

hvr commented Jan 23, 2019

@michaelpj you seem to have tried with the legacy v1 commands; I tried with v2-build instead:

$ cabal v2-build -w ghc-8.4.4
Resolving dependencies...
Build profile: -w ghc-8.4.4 -O1
In order, the following will be built (use -v for more details):
 - test-0.1.0.0 (lib) (first run)
 - test-0.1.0.0 (lib:irritant) (first run)
Configuring library for test-0.1.0.0..
Configuring library 'irritant' for test-0.1.0.0..
Warning: Packages using 'cabal-version: >= 1.10' must specify the
'default-language' field for each component (e.g. Haskell98 or Haskell2010).
If a component uses different languages in different modules then list the
other ones in the 'other-languages' field.
Warning: Packages using 'cabal-version: >= 1.10' must specify the
'default-language' field for each component (e.g. Haskell98 or Haskell2010).
If a component uses different languages in different modules then list the
other ones in the 'other-languages' field.
Preprocessing library 'irritant' for test-0.1.0.0..
Building library 'irritant' for test-0.1.0.0..
Preprocessing library for test-0.1.0.0..
Building library for test-0.1.0.0..
[1 of 1] Compiling Test             ( src/Test.hs, /tmp/cabal-bug-package-db/dist-newstyle/build/x86_64-linux/ghc-8.4.4/test-0.1.0.0/build/Test.o )

$ ghc-8.4.4 Use.hs
Loaded package environment from /tmp/cabal-bug-package-db/.ghc.environment.x86_64-linux-8.4.4

$ echo $?
0

@michaelpj
Copy link
Collaborator Author

Unfortunately cabal new-build is still unusable for me.

I did manage to solve my original problem by using a cabal command that set up the paths correctly for me (in this case cabal doctest), but I thought it was surprising that when I tried to do it manually with the package db it didn't work.

Feel free to close if you're not interested in fixing this.

@hvr
Copy link
Member

hvr commented Jan 23, 2019

@michaelpj it's not that we aren't interested, it's rather that the highest priority currently has getting v2-build ready to replace old-build. You say

Unfortunately cabal new-build is still unusable for me.

and since we're going to make it default real-soon it would be great if you can elaborate what is missing so we can try to address that... :-)

@michaelpj
Copy link
Collaborator Author

Ah sorry, I interpreted that as "we're probably not interested in fixing v1 stuff" which would be reasonable :)

I don't think it's anything you don't have tickets for. For my organization at least the following are blockers:

@michaelpj
Copy link
Collaborator Author

Hm, I'm observing this also in the package databases produced by Nix. I don't have a minimal reproduction yet, but it's something like:

  • Cabal 2.2.0.0
  • ghc 8.4.4+ (not observable with ghc 8.4.3)

In this case the combined package db for my dependencies has the package listed as present when I do ghc-pkg, but not when I actually try to compile something.

However, I think I have found a clue:

  • When I run cabal repl or similar I see it passing -package-id <package name>-<long hash for all the dependencies.
  • If I run ghc -package-id <package-name>-<long-hash> (which I pulled from out of the .conf file in the package db) then it does work.

Now, the docs for -package-id say that it exposes the package with the given id. But the .conf file in the package says it's exposed already, and ghc-pkg expose does nothing.

The only thing I can think of that's consistent with all this is that GHC disagrees with ghc-pkg and cabal about whether a package is exposed or not, and it requires to be told again that it should be exposed.

@michaelpj
Copy link
Collaborator Author

Further evidence that it's just looking for the wrong package id: the error message I get in both cases is not the one you get when you try to import from a hidden package, so it's not the exposure status that's wrong.

@michaelpj
Copy link
Collaborator Author

Correction:

  • In the original case I reported I get a "cannot import module from hidden package" error
  • In the second issue I get a generic "could not find module error"

Both are mitigated either by passing -package-id or just plain -package!

@michaelpj
Copy link
Collaborator Author

I opened a trac ticket for this since it seems like ghc-pkg and ghc are disagreeing.

@michaelpj
Copy link
Collaborator Author

Returning to this, I have a note about new-build: this works with new-build because the environment file specifies all the package-ids manually, regardless of whether they're exposed in the package db. I think this masks the bug, since it means we don't notice that a package that should be exposed is in fact hidden.

@gbaz gbaz added the type: bug label Aug 28, 2021
MangoIV pushed a commit to MangoIV/plutus-contract that referenced this issue Sep 3, 2022
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer. We can then organize
things a bit more neatly inside.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
@andreasabel andreasabel added the re: internal library Concerning internal libraries in packages label Feb 4, 2023
kayvank pushed a commit to input-output-hk/marconi that referenced this issue Mar 11, 2023
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer. We can then organize
things a bit more neatly inside.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
kayvank pushed a commit to input-output-hk/marconi that referenced this issue Mar 22, 2023
The documentation for `addCorePlugin` says it has to be in a separate
package to the plugin. However it appears that in GHC's eyes internal
libraries in a Cabal package are separate packages, so we can hide all
of this in a single package, which is much nicer. We can then organize
things a bit more neatly inside.

We then have to do some shennanigans to get doctest to work. The normal
tests work fine, since Cabal registers the built internal library as a
package, and that's fine. But doctest interprets things, which *doesn't*
work.

Now, I initially tried to just pass the appropriate `-package-db`
arguments to `doctest`, but in doing so I hit what appears to be a [Cabal
bug with internal libraries](haskell/cabal#5857).
However, somehow `cabal repl` and friends seem to get around this, and
fortunately there is now a `cabal doctest` command, which amazingly
also seems to pass the right set of obscure flags. So we use that.

Finally, in an extremely irritating discovery, apparently [Haddock
doesn't work with internal libraries]
(haskell/cabal#4969) (possibly fixed in a
newer Cabal, who knows), so I ended up having to exclude `plutus-tx`
from Haddock generation for now.
@michaelpj
Copy link
Collaborator Author

I fixed this in GHC.

@Mikolaj
Copy link
Member

Mikolaj commented Mar 27, 2023

Well done! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
re: internal library Concerning internal libraries in packages type: bug
Projects
None yet
Development

No branches or pull requests

5 participants