Skip to content
This repository has been archived by the owner on Dec 2, 2024. It is now read-only.

Commit

Permalink
Use Cardano Haskell package repository ("CHaP") (#797)
Browse files Browse the repository at this point in the history 
* Switch to CHaP versions where possible

* Add comment

* Remove cardano-config

* Annontate package names

* Bump index-state

* Rework CONTRIBUTING.adoc

* Bump hackage.nix too

* Remove unused packages

* Remove outdated commit

* Constraints work

* No-op

* Still works

* Still works

* cabal-fmt

* Updated CONTRIBUTING.adoc and doc/BRANCHING-AND-RELEASE.adoc

* Fix all the constraints

* Remove cardano-node srp once again

* Added more comments, fixed formatting
  • Loading branch information
@andreabedini
andreabedini authored Nov 21, 2022
1 parent 12ccca8 commit 7d3687d
Show file tree
Hide file tree
Showing 9 changed files with 135 additions and 256 deletions.
83 changes: 45 additions & 38 deletions CONTRIBUTING.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -108,55 +108,61 @@ NOTE: you must run your editor from the nix shell, and not from, say, an
application launcher like spotlight or dmenu for your editor to find it in
its PATH.

=== How to add a new Haskell package

You need to do a few things when adding a new package, in the following order:

. Add the cabal file for the new package.
. Add the package to link:cabal.project[`cabal.project`].
. Add the package to link:stack.yaml[`stack.yaml`].
. Check that you can run `nix build -f default.nix plutus-apps.haskell.projectPackages.<package name>` successfully.

[[update-haskell-pins]]
=== How to update our pinned Haskell dependencies
[[update-index-state]]
=== Updating dependencies

We have pinned versions of some Haskell packages specified via the usual `source-repository-package` (Cabal) and `extra-dep` (Stack) mechanisms.
==== ... from Hackage

These can be managed normally, but ensure that:
Updating package dependencies from Hackage should work like normal in a
Haskell project. The most important thing to note is that we pin the
``index-state`` of the Hackage package index in ``cabal.project``. This
means that cabal will always see Hackage “as if” it was that time, ensuring
reproducibility. But it also means that if you need a package version that
was released *after* that time, you need to bump the ``index-state`` (and
to run ``cabal update`` locally). Please also note that index-state is a
property of your working environment (the cabal project) not of the
packages. This means that downstream consumers have no idea of your
``index-state`` setting and you need to make sure your packages work
correctly also without ``index-state``.

* If it is an `source-repository-package`/`extra-dep` from Git, you update the `sha256` mapping in `nix/haskell.nix`.
For the moment you have to do this by hand, using the following command to get the sha: `nix-prefetch-git --quiet <repo-url> <rev> | jq .sha256`, or by just getting it wrong and trying to build it, in which case Nix will give you the right value.
* For certain dependencies like `plutus-apps`, please run `nix flake lock --update-input <input-name>` to update `flake.lock`. A wrong-hash error on a `nix build` is a common hint.
Because of how we use Nix to manage our Haskell build, whenever you do this
you will also need to pull in the Nix equivalent of the newer
``index-state``. You can do this by running ``nix flake lock --update-input
hackage-nix``.

[[update-nix-pins]]
=== How to update our pinned Nix dependencies
==== ... from the Cardano package repository

We pin versions of some git repositories that are used by Nix, for example `nixpkgs`.
Many Cardano packages are not on Hackage and are instead in the `Cardano
package repository
<https://github.com/input-output-hk/cardano-haskell-packages>`, see the
README for (lots) more information.

We use Nix flakes to manage these dependencies, even though we do not
actually use flakes for normal Nix builds. To manage the dependencies,
see https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-flake.html#flake-inputs[the Nix flake inputs documentation]
and https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-flake-lock.html[the Nix flake lock command].
Getting new packages from there works much like getting them from Hackage.
The differences are that it can have an independent ``index-state``, and
that there is a different Nix command you need to run afterwards: ``nix flake
lock --update-input CHaP``.

Specifically, you will probably want to say `nix flake lock --update-input <input-name>`.
Do *not* use `nix flake update`, as that will update all the inputs, which we typically don't want to do.
===== Using unreleased versions of dependencies

Inside of the project nix-shell, a version of Nix with flakes is available under the alias `nix-flakes`.
Sometimes we need to use an unreleased version of one of our dependencies,
either to fix an issue in a package that is not under our control, or to
experiment with a pre-release version of one of our own packages.

[[update-index-state]]
=== How to update the Hackage index state
You can use a ``source-repository-package`` stanza to pull in the
unreleased version.

The Hackage index state is pinned to a particular time in `cabal.project`.
This helps with reproducibility: alongside using the same version of `cabal`, this ensures that everyone will get the same result from the `cabal` version solver.
If you want to use a Hackage package from after the pinned index state time, you need to bump it.
This is not a big deal, since all it does is change what packages `cabal` considers to be available when doing solving, but it *may* result in different versions being picked, so it's not completely safe.
Please note that consumers of our packages will not pull these unreleased
versions when compiling our packages so consider using ``source-repository-package``
only as a temporary solution.

Note that `cabal` itself keeps track of what index states it knows about, so you may need to update this with `cabal update` in order for `cabal` to be happy.
For packages that we do not control, we can end up in a situation where we
have a fork that looks like it will be long-lived or permanent (e.g. the
maintainer is unresponsive, or the change has been merged but not
released).

The Nix code which builds our packages also cares about the index state.
The set of index states which it knows about is controlled by `hackage.nix`, which is a Nix representation of Hackage.
This therefore needs to be newer than the index state.
You can update it xref:update-nix-pins[with the Nix flake commands].
In that case, release a patched version to the `Cardano package repository
<https://github.com/input-output-hk/cardano-haskell-packages>`, which
allows us to remove the ``source-repository-package`` stanza.

=== How to work with a local copy of source dependencies

Expand Down Expand Up @@ -350,7 +356,7 @@ Do not mix business logic with such things as reformatting and refactoring in a
A pull request is communication, so as usual, put yourself in the position of the reader: what does your audience (the reviewer) need to know to do their job?
This information is easy for you to access, but hard for them to figure out, so write it down!

However, better to put information in the code, commit messages, or changelog if possible: these persist but PR descriptions do not.
However, better to put information in the code, commit messages or changelog if possible: these persist but PR descriptions do not.
It's okay to repeat information from such places, or simply to point to it.
For one-commit PRs, Github will automatically populate the PR description with the commit message, so if you've written a good commit message you're done!
Sometimes there is "change-related" information that doesn't belong in a commit message but is useful ("Kris I think this will fix the issue you had yesterday").
Expand Down Expand Up @@ -474,6 +480,7 @@ Usually we don't edit `CHANGELOG.md` directly.
You can make a changelog fragment using `scriv create` in the package directory, but you can also just create the fragment directly with an editor.
A fragment is a markdown file beginning with a header giving the category of change.


== Supporting systems

=== Continuous integration
Expand Down
252 changes: 57 additions & 195 deletions cabal.project
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
-- Bump this if you need newer packages.
index-state: 2022-09-29T00:00:00Z
-- Custom repository for cardano haskell packages
-- See https://github.com/input-output-hk/cardano-haskell-packages on how to use CHaP in a Haskell project.
repository cardano-haskell-packages
url: https://input-output-hk.github.io/cardano-haskell-packages
secure: True
root-keys:
3e0cce471cf09815f930210f7827266fd09045445d65923e6d0238a6cd15126f
443abb7fb497a134c343faf52f0b659bd7999bc06b7f63fa76dc99d631f9bea1
a86a1f6ce86c449c46666bda44268677abf29b5b2d2eb5ec7af903ec2f117a82
bcec67e8e99cabfa7764d75ad9b158d72bfacf70ca1d0ec8bc6b4406d1bf8413
c00aae8461a256275598500ea0e187588c35a5d5d7454fb57eac18d9edb86a56
d4a35cd3121aa00d18544bb0ac01c3e1691d618f462c46129271bccf39f7e8ee

-- See CONTRIBUTING.adoc for how to update index-state
index-state: 2022-11-14T00:20:02Z

-- See CONTRIBUTING.adoc for how to update index-state
index-state:
, hackage.haskell.org 2022-11-14T00:20:02Z
, cardano-haskell-packages 2022-11-17T04:56:26Z

packages: cardano-streaming
doc
Expand Down Expand Up @@ -35,18 +53,40 @@ benchmarks: true
-- 'tasty' output.
test-show-details: direct

-- cardano-addresses unit tests bring in some version constraint conflicts:
--
-- * it has strict aeson < 1.5 dep - this will be fixed in the next release.
allow-newer:
*:aeson
, size-based:template-haskell
-- cardano-ledger packages need aeson >2, the following packages have a
-- too restictive upper bounds on aeson, so we relax them here. The hackage
-- trustees can make a revision to these packages cabal file to solve the
-- issue permanently.
, ekg:aeson
, ekg-json:aeson
, openapi3:aeson
, servant:aeson
, servant-client-core:aeson
, servant-server:aeson

constraints:
aeson >= 2
, hedgehog >= 1.1
, protolude < 0.3.1
, algebraic-graphs < 0.7
-- cardano-prelude-0.1.0.0 needs
, protolude <0.3.1

-- cardano-ledger-byron-0.1.0.0 needs
, cardano-binary <1.5.0.1

-- plutus-core-1.0.0.1 needs
, cardano-crypto-class >2.0.0.0
, algebraic-graphs <0.7

-- cardano-ledger-core-0.1.0.0 needs
, cardano-crypto-class <2.0.0.1

-- cardano-crypto-class-2.0.0.0.1 needs
, cardano-prelude <0.1.0.1

-- dbvar from cardano-wallet needs
, io-classes <0.3.0.0

-- newer typed-protocols need io-classes>=0.3.0.0 which is incompatible with dbvar's constraint above
, typed-protocols==0.1.0.0

-- The plugin will typically fail when producing Haddock documentation. However,
-- in this instance you can simply tell it to defer any errors to runtime (which
Expand Down Expand Up @@ -136,78 +176,19 @@ source-repository-package
location: https://github.com/sevanspowell/hw-aeson
tag: b5ef03a7d7443fcd6217ed88c335f0c411a05408

-- Using a fork until our patches can be merged upstream
source-repository-package
type: git
location: https://github.com/input-output-hk/optparse-applicative
tag: 7497a29cb998721a9068d5725d49461f2bba0e7a

-- Should follow cardano-wallet.
source-repository-package
type: git
location: https://github.com/input-output-hk/Win32-network
tag: 3825d3abf75f83f406c1f7161883c438dac7277d

-- Should follow cardano-wallet.
source-repository-package
type: git
location: https://github.com/input-output-hk/cardano-addresses
tag: b7273a5d3c21f1a003595ebf1e1f79c28cd72513
subdir: command-line
core

-- Should follow cardano-wallet.
source-repository-package
type: git
location: https://github.com/input-output-hk/cardano-base
tag: cc049d7c9b9a0129c15b1355fd1dff9e1a1a551c
subdir:
base-deriving-via
binary
binary/test
cardano-crypto-class
cardano-crypto-praos
cardano-crypto-tests
orphans-deriving-via
measures
strict-containers
slotting
-- cardano-addresses-cli
command-line
-- cardano-addresses
core

-- Should follow cardano-wallet.
source-repository-package
type: git
location: https://github.com/input-output-hk/cardano-crypto
tag: f73079303f663e028288f9f4a9e08bcca39a923e

-- Should follow cardano-node.
-- But in case there are failures with the plutus version, update to the latest
-- commit hash of the release/1.0.0 plutus branch.
source-repository-package
type: git
location: https://github.com/input-output-hk/plutus
tag: d5f6f4a5505094490f61c8d164515adf7c8f46cc
subdir:
plutus-core
plutus-ledger-api
plutus-tx
plutus-tx-plugin
prettyprinter-configurable
stubs/plutus-ghc-stub
word-array

-- Should follow cardano-node
source-repository-package
type: git
location: https://github.com/input-output-hk/ekg-forward
tag: 297cd9db5074339a2fb2e5ae7d0780debb670c63

-- Should follow cardano-node
source-repository-package
type: git
location: https://github.com/input-output-hk/cardano-config
tag: 1646e9167fab36c0bff82317743b96efa2d3adaa

-- Should follow cardano-wallet.
-- This is needed because we rely on an unreleased feature
-- https://github.com/input-output-hk/cardano-ledger/pull/3111
source-repository-package
type: git
location: https://github.com/input-output-hk/cardano-ledger
Expand Down Expand Up @@ -236,122 +217,3 @@ source-repository-package
libs/small-steps
libs/small-steps-test
libs/non-integral

-- Should follow cardano-wallet.
-- More precisally, this should be a version compatible with the current
-- Cardano mainnet (>=1.35).
source-repository-package
type: git
location: https://github.com/input-output-hk/cardano-node
tag: 1.35.4
subdir:
cardano-api
cardano-git-rev
cardano-cli
cardano-node
cardano-submit-api
cardano-testnet
trace-dispatcher
trace-resources
trace-forward

-- Should follow cardano-wallet.
source-repository-package
type: git
location: https://github.com/input-output-hk/cardano-prelude
tag: bb4ed71ba8e587f672d06edf9d2e376f4b055555
subdir: cardano-prelude
cardano-prelude-test

-- Should follow cardano-wallet.
source-repository-package
type: git
location: https://github.com/input-output-hk/goblins
tag: cde90a2b27f79187ca8310b6549331e59595e7ba

-- Direct dependency.
-- Are you thinking of updating this tag to some other commit?
-- Please ensure that the commit you are about to use is the latest one from
-- the *develop* branch of this repo:
-- * <https://github.com/input-output-hk/iohk-monitoring-framework/commits/develop>
-- (not master!)
--
-- In particular we rely on the code from this PR:
-- * <https://github.com/input-output-hk/iohk-monitoring-framework/pull/622>
-- being merged.
source-repository-package
type: git
location: https://github.com/input-output-hk/iohk-monitoring-framework
tag: 066f7002aac5a0efc20e49643fea45454f226caa
subdir: contra-tracer
iohk-monitoring
plugins/backend-aggregation
plugins/backend-ekg
plugins/backend-monitoring
plugins/backend-trace-forwarder
plugins/scribe-systemd
tracer-transformers

-- Should follow cardano-wallet.
source-repository-package
type: git
location: https://github.com/input-output-hk/ouroboros-network
tag: b6ab4ca226ba541ee89374a30824351a91285226
subdir:
monoidal-synchronisation
network-mux
ouroboros-consensus
ouroboros-consensus-byron
ouroboros-consensus-cardano
ouroboros-consensus-protocol
ouroboros-consensus-shelley
ouroboros-network
ouroboros-network-framework
ouroboros-network-testing
ntp-client

-- Should follow cardano-node.
source-repository-package
type: git
location: https://github.com/input-output-hk/io-sim
tag: 57e888b1894829056cb00b7b5785fdf6a74c3271
subdir:
io-classes
io-sim
strict-stm

source-repository-package
type: git
location: https://github.com/input-output-hk/moo
tag: 8c487714fbfdea66188fcb85053e7e292e0cc348

-- Should follow cardano-node.
source-repository-package
type: git
location: https://github.com/input-output-hk/typed-protocols
tag: 181601bc3d9e9d21a671ce01e0b481348b3ca104
subdir:
typed-protocols
typed-protocols-cborg
typed-protocols-examples

-- Should follow plutus.
-- https://github.com/Quid2/flat/pull/22 fixes a potential exception
-- when decoding invalid (e.g. malicious) text literals.
source-repository-package
type: git
location: https://github.com/Quid2/flat
tag: ee59880f47ab835dbd73bea0847dab7869fc20d8

-- Should follow cardano-wallet.
-- Until https://github.com/tibbe/ekg-json/pull/12 gets merged with aeson2 support
source-repository-package
type: git
location: https://github.com/vshabanov/ekg-json
tag: 00ebe7211c981686e65730b7144fbf5350462608

-- Should follow cardano-wallet
source-repository-package
type: git
location: https://github.com/input-output-hk/hedgehog-extras
tag: 714ee03a5a786a05fc57ac5d2f1c2edce4660d85
Loading

0 comments on commit 7d3687d

Please sign in to comment.