Support non-Hackage dependencies

[23Skidoo]

Summary by @ezyang:

• Syntactic support for these URLs exists: in the packages field you can specify remote tarballs or local tarballs, and using the source-repository syntax (from Cabal file) you can point to a VCS URL (this parser should also have been reused for cabal.project)
• Support for downloading from various places exists. There's support in the old-build path for local/remote tarballs. cabal get -s supports getting things from source repositories.
• However, these are not used for anything: the internals need to be plumbed up to use this. Look at old stuff and determine how to port it over.

This should be doable in a day.

---

This came up on Twitter:

https://twitter.com/darinmorrison/status/527716059889954816

Basically, people want to publish packages that depend on stuff that is not on Hackage. One solution is to upload a forked package (example), but this has its own downsides (Hackage namespace becomes polluted). Implementing something like build-depends: https://repo.git@revision would be a nice alternative. Most other package managers (e.g. sbt) support this.

It should also be possible to prohibit downloading packages not on Hackage via a config file option.

See also #1534.

[mietek]

Halcyon supports declaring sandbox sources with the HALCYON_SANDBOX_SOURCES option.

See Try Haskell for an example of declaring the git HEAD version of mueval as a sandbox source.

[winterland1989]

I'd like to bump this issue's priority and start to discuss implementation, currently it's stopping us using cabal in company environment because we have multiple packages can't be upload to hackage, and a hackage mirror is an overkill. now we use stack to solve this headache, but it's really not necessary to rely both on hackage and stackage infrastructure.

[ezyang]

Hello @winterland1989. Have you tried the new new-build functionality (http://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-local-builds/). If you have all of the non-Hackagable packages on disk in a build tree, you can write a cabal.project to pick them up, and then cabal new-build will automatically build them together. Does that solve your proximal problem?

[winterland1989]

Thanks for your reply @ezyang ! I'm aware that there're multiple ways solving local package discovery: cabal sandbox add-source or a cabal.project file both solve this, and i'm aware that cabal.project file support parsing URI as package now. But we need an automatic way to pull/update the remote file into disk without hassle, currently available options are git submodule, hand-role shell and stack.(you really want to automate this process when you got more than 5 local depends).

If cabal.project file is the final solution, i'll dig into it to see what i can do.

[23Skidoo]

@winterland1989
There is already some code for declaring VCS dependencies in cabal.project files: https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectConfig.hs#L523

If you want to contribute code to Cabal, I suggest improving the nix-local-build code path, since the plan is to make it the default in the future.

[alanz]

I would like to see something like http://docs.haskellstack.org/en/stable/yaml_configuration/#complex-package-locations-location

[phadej]

Looks like we only need to teach cabal-install how to fetch stuff from the internet:

cabal/cabal-install/Distribution/Client/ProjectConfig.hs

Line 745 in 5423d77

fail $ "TODO: add support for fetching and reading local tarballs, remote "

tar support shouldn't be difficult, as cabal handles it already. git is trickier, as we need to teach cabal-install to call git clone (and darcs).

Question: were to put downloaded stuff, dist-newstyle/downloaded/hash-of-the-source ?

[phadej]

(For now one can use git submodule, effectively the result should be the same)

[alanz]

FYI, every github commit can be downloaded as a zip file, e.g.

https://github.com/haskell/cabal/archive/master.zip
https://github.com/haskell/cabal/archive/59eabde01e7df7ac51602b40d867fe5ece2992c2.zip

[winterland1989]

FYI, every github commit can be downloaded as a zip file, e.g.

Yes, but that's github's convention, we probably shouldn't rely on that convention. In fact we can safely assume user have git binary installed if they want to use git depends, and we can use process package manage git clone/pull.

[alanz]

A use case for this is during development when another client component has not yet been released to hackage. E.g. ghc-mod for HaRe. This is a temporary state.

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

[alanz]

And there is a general-purpose variant too

https://git-scm.com/docs/git-archive

This supports a --remote=XXX flag

[phadej]

@winterland1989 I hope we can interpret @alanz comment, as so even support for remote tarballs would make him a lot happier :)

[winterland1989]

Yes, support for remote tarballs is enough for me too ; )

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

This is also my main motivation for this feature requesting, manually manage forked source is frustrating.

[ezyang]

If you do this for nix-local-build, a determination of whether or not such external dependencies should be local or external needs to be made. If they're treated as local (which I believe is what the current code will do if you list it in packages or similar fields), then they are not eligible to be cached and reused in the external store. That's probably not desirable.

[dcoutts]

In principle if we can get a reliable source hash (and guarantee this isn't changed locally) then we can use the store approach. So for git we can use the git hash, but we'd have to avoid letting users fiddle with the sources locally in that case. If we allow fiddling then it has to be treated like a local dir.

[23Skidoo]

If we allow fiddling then it has to be treated like a local dir.

We can ask git if the source has been fiddled with and then fall back to treating it like a local dir if it is the case.

[ezyang]

I'm inclined to just not allow specification of a hash if you don't specify a remote URL. Don't really want new-build to be checking out git branches or ignoring my hash...

[bitemyapp]

I'm looking forward to celebrating this issue's second 🎂 in two days.

[bitemyapp]

Happy 🎂!

[gbaz]

The provenance-qualified imports proposal (ghc-proposals/ghc-proposals#115) is relevant here.

[hvr]

It's a pity @bitemyapp won't get to celebrate schadenfreude this year... ;-)

[typedrat]

Closed by #5351.

[nomeata]

Exciting! I want to play with this

But is this really fixed? https://github.com/haskell/cabal/blob/master/Cabal/doc/nix-local-build.rst on master still says

    3. [STRIKEOUT:They can specify an ``http``, ``https`` or ``file``
       URL, representing the path to a remote tarball to be downloaded
       and built.] (not implemented yet)

and

[STRIKEOUT:There is also a stanza source-repository-package for specifying packages from an external version control.] (Not implemented.)

[typedrat]

That's a documentation bug, I believe.

[nomeata]

I will know in a few minutes when cabal has built and I can try the feature :-)
…
Yes, seems to work. Great!