From 2614759f076c0cbb73c0d5d7c7a8aced718bcfac Mon Sep 17 00:00:00 2001 From: Kosyrev Serge Date: Tue, 23 Aug 2022 19:58:53 +0800 Subject: [PATCH] README: update development section --- README.rst | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 122 insertions(+), 2 deletions(-) diff --git a/README.rst b/README.rst index 07d212cb5a5..ea8004d41c0 100644 --- a/README.rst +++ b/README.rst @@ -416,7 +416,127 @@ Byron vote submission: Development **** -Development tooling +Overview +==== + +The ``cardano-node`` development is primarily based on the Nix infrastructure (https://nixos.org/ ), which enables packaging, CI, development environments and deployments. + +On how to set up Nix for ``cardano-node`` development, please see `Building Cardano Node with nix `_. + +Workbench: a local cluster playground +==== + +You can quickly spin up a local cluster, based on any of a wide variety of configurations, and put it under a transaction generation workload -- using the ``workbench`` environment: + +1. Optional: choose a workbench profile: + - ``default`` stands for a light-state, 6-node cluster, under saturation workload, indefinite runtime + - ``ci-test`` is the profile run in the node CI -- very light, just two nodes and short runtime + - ``devops`` is an unloaded profile (no transaction generation) with short slots -- ``0.2`` sec. + - ..and many more -- which can be either: + - listed, by ``make ps`` + - observed at their point of definition: `nix/workbench/profiles/prof1-variants.jq `_ +2. Optional: select mode of operation, by optionally providing a suffix: + - default -- no suffix -- just enter the workbench shell, allowing you to run ``start-cluster`` at any time. Binaries will be built locally, by ``cabal``. + - ``autostay`` suffix -- enter the workbench shell, start the cluster, and stay in the shell afterwards. Binaries will be built locally, by ``cabal``. + - ``autonix`` suffix -- enter the workbench shell, start the cluster. All binaries will be provided by the Nix CI. + - ..there are other modes, as per `lib.mk `_ +3. Enter the workbench shell for the chosen profile & mode: + ``make `` or ``make -`` (when there is a suffix). +4. Optional: start cluster: + Depending on the chosen mode, your cluster might already start, or you are expected to start it yourself, using ``start-cluster``. + +The workbench services are available only inside the workbench shell. + +Using Cabal +---- + +By default, all binaries originating in the ``cardano-node`` repository are available to ``cabal build`` and ``cabal run``, unless the workbench was entered using one of the pure ``*nix`` modes. Note that in all cases, the dependencies for the workbench are supplied though Nix and have been built/tested on CI. + +**Dependency localisation** -or- *Cabal&Nix for painless cross-repository work* +---- + +Normally, to work on ``cardano-node`` dependencies in the context of the node itself, one needs to go through an expensive multi-step process, mandated by the Nix model: + - Make changes to a dependency (say ``ouroboros-consensus-cardano`` in the ``ouroboros-network`` repository), commit them & push to Github. + - Calculate & bump the pin for that dependency in ``cardano-node``, inside its ``cabal.project``. + - Re-enter the Nix shell in ``cardano-node``, suffering full rebuilds of everything. + +The **dependency localisation** workflow allows us to pick a subset of leaf dependencies of the ``cardano-node`` repository, and declare them *local* -- so they can be iterated upon using the ``cabal build`` / ``cabal run`` of ``cardano-node`` itself. This cuts development iteration time dramatically and enables effective cross-repo development of the full stack of Cardano packages. + +Without further ado (**NOTE**: *the order of steps is important!*): + +1. Ensure that your ``cardano-node`` checkout is clean, with no local modifications. Also, ensure that you start outside the node's Nix shell. +2. Check out the repository with the dependencies, *beside* the ``cardano-node`` checkout. You have to check out the git revision of the dependency used by your ``cardano-node`` checkout -- as listed in ``cardano-node/cabal.project``. + - we'll assume the ``ouroboros-network`` repository + - ..so a certain parent directory will include checkouts of both ``ouroboros-network`` and ``cardano-node``, at the same level + - ..and the git revision checked out in ``ouroboros-network`` will match the corresponding ``source-repository-package`` clause in ``cardano-node/cabal.project``. + - Extra point #1: you can localise/check out several dependency repositories + - Extra point #2: for the dependencies that are not listed in ``cabal.project`` of the node -- how do you determine the version to check out? You can ask the workbench shell: + 1. Temporarily enter the workbench shell + 2. Look for the package version in ``ghc-pkg list`` + 3. Use that version to determine the git revision of the dependency's repository (using a tag or some special knowledge about the version management of said dependency). +3. Enter the workbench shell, as per instructions in previous sections -- or just a plain Nix shell. +4. Ensure you can build ``cardano-node`` with Cabal: ``cabal build exe:cardano-node``. If you can't something else is wrong. +5. Determine the *leaf dependency set* you will have to work on. The *leaf dependency set* is defined to include the target package you want to modify, and its reverse dependencies -- that is, packages that depend on it (inside the dependency repository). + - let's assume, for example, that you want to modify ``ouroboros-consensus-shelley`` + - ``ouroboros-consensus-shelley`` is not a leaf dependency in itself, since ``ouroboros-consensus-cardano`` (of the same ``ouroboros-network`` repository) depends on it -- so the *leaf dependency set* will include both of them. + - you might find out that you have to include a significant fraction of packages in ``ouroboros-network`` into this *leaf dependency set* -- do not despair. +6. Edit the ``cardano-node/cabal.project`` as follows: + - for the *leaf dependency set* + 1. in the very beginning of the ``cabal.project``, add their relative paths to the ``packages:`` section, e.g.: + .. code-block:: console + + packages: + cardano-api + cardano-cli + ... + trace-resources + trace-forward + ../ouroboros-network/ouroboros-consensus-shelley + ../ouroboros-network/ouroboros-consensus-cardano + + 2. in the corresponding ``source-repository-package`` section, comment out mentions of the packages, e.g.: + .. code-block:: console + + source-repository-package + type: git + location: https://github.com/input-output-hk/ouroboros-network + tag: c764553561bed8978d2c6753d1608dc65449617a + --sha256: 0hdh7xdrvxw943r6qr0xr4kwszindh5mnsn1lww6qdnxnmn7wcsc + subdir: + monoidal-synchronisation + network-mux + ouroboros-consensus + ouroboros-consensus-byron + -- ouroboros-consensus-cardano + ouroboros-consensus-protocol + -- ouroboros-consensus-shelley + ouroboros-network + ouroboros-network-framework + ouroboros-network-testing + +7. The two packages have now became **local** -- when you try ``cabal build exe:cardano-node`` now, you'll see that Cabal starts to build these dependencies you just localised. Hacking time! + +Hoogle +---- + +The workbench shell provides ``hoogle``, with a local database for the full set of dependencies: + +.. code-block:: console + + [nix-shell:~/cardano-node]$ hoogle search TxId + Byron.Spec.Ledger.UTxO newtype TxId + Byron.Spec.Ledger.UTxO TxId :: Hash -> TxId + Cardano.Chain.UTxO type TxId = Hash Tx + Cardano.Ledger.TxIn newtype TxId crypto + Cardano.Ledger.TxIn TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto + Cardano.Ledger.Shelley.API.Types newtype TxId crypto + Cardano.Ledger.Shelley.API.Types TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto + Cardano.Ledger.Shelley.Tx newtype TxId crypto + Cardano.Ledger.Shelley.Tx TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto + Ouroboros.Consensus.HardFork.Combinator data family TxId tx :: Type + -- plus more results not shown, pass --count=20 to see more + +Supplementary tooling ==== GHCID @@ -428,7 +548,7 @@ Haskell Language Server ---- When using Haskell Language Server with Visual Studio Code, you may find that -`HLINT annotations are ignored`. +`HLINT annotations are ignored `_. To work around this, you may run the script ``./scripts/reconfigure-hlint.sh`` to generate a ``.hlint.yaml`` file with HLINT ignore rules derived from the source code.