Skip to content

Commit

Permalink
Merge #4377
Browse files Browse the repository at this point in the history 
4377: README:  document package localisation r=deepfire a=deepfire

In the README:

1. Add a ToC.
2. Fix the structure, so the ToC actually makes sense.
3. Fix some minor RST syntax issues.
4. Some very minor fixes.
5. Expand the Development section with the workbench usage info.
6. Add `cabal-plan` and `graphmod` to the workbench shell.
7. Makes the node start on Darwin, when metric collection is enabled, by fixing #4380

Co-authored-by: Kosyrev Serge <[email protected]>
Co-authored-by: Michael Karg <[email protected]>
  • Loading branch information
@iohk-bors @deepfire @mgmeier
3 people authored Aug 24, 2022
2 parents e409603 + f75b4b3 commit e36c1a7
Show file tree
Hide file tree
Showing 3 changed files with 179 additions and 50 deletions.
225 changes: 176 additions & 49 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,10 @@
</tr>
</table>

.. contents:: Contents

*************************
``cardano-node`` Overview
Overview of the ``cardano-node`` repository
*************************

Integration of the `ledger <https://github.com/input-output-hk/cardano-ledger-specs>`_, `consensus <https://github.com/input-output-hk/ouroboros-network/tree/master/ouroboros-consensus>`_,
Expand All @@ -39,8 +41,12 @@ Network Configuration, Genesis and Topology Files

The latest supported networks can be found at `<https://book.world.dev.cardano.org/environments.html>`_

How to build
============
****
Obtaining ``cardano-node``
****

Building from source
====

Documentation for building the node can be found `here <https://docs.cardano.org/getting-started/installing-the-cardano-node>`_.

Expand All @@ -54,15 +60,15 @@ You can download the latest version of ``cardano-node`` and ``cardano-cli``:
* `macos <https://hydra.iohk.io/job/Cardano/cardano-node/macos.cardano-node-macos/latest-finished>`_

Windows Executable
==================
----

Download
--------
++++

You can download `here <https://hydra.iohk.io/job/Cardano/cardano-node/linux.windows.cardano-node-win64/latest-finished>`_.

Instructions
------------
++++

The download includes cardano-node.exe and a .dll. To run the node with cardano-node run you need to reference a few files and directories as arguments. These can be copied from the cardano-node repo into the executables directory. The command to run the node on mainnet looks like this:

Expand All @@ -79,8 +85,12 @@ You can pull the docker image with the latest version of cardano-node from `here
docker pull inputoutput/cardano-node
``cardano-node``
================
****
Using ``cardano-node``
****

Command line summary: ``cardano-node``
====
This refers to the client that is used for running a node.

The general synopsis is as follows:
Expand Down Expand Up @@ -128,37 +138,26 @@ The general synopsis is as follows:

* ``--validate-db`` - Flag to revalidate all on-disk database files

Configuration ``.yaml`` files
=============================
Configuration
====

The ``--config`` flag points to a ``.yaml`` file that is responsible to configuring the logging & other important settings for the node. E.g. see the Byron mainnet configuration in this
The ``--config`` flag points to a ``.yaml`` (or a structurally equivalent ``.json``) file that is responsible to configuring the logging & other important settings for the node. E.g. see the Byron mainnet configuration in this
`configuration.yaml <https://github.com/input-output-hk/cardano-node/blob/master/configuration/defaults/byron-mainnet/configuration.yaml>`_.

Some of the more important settings are as follows:

* ``Protocol: RealPBFT`` -- Protocol the node will execute

* ``RequiresNetworkMagic``: RequiresNoMagic -- Used to distinguish between mainnet (``RequiresNoMagic``) and testnets (``RequiresMagic``)


Logging
========

Logs are output to the ``logs/`` dir.

Profiling & statistics
======================

Profiling data and RTS run stats are stored in the ``profile/`` dir.

Please see ``scripts/README.md`` for how to obtain profiling information using the scripts.

Scripts
=======

Please see ``scripts/README.md`` for information on the various scripts.

``cardano-cli``
===============
****
Using ``cardano-cli``
****

A CLI utility to support a variety of key material operations (genesis, migration, pretty-printing..) for different system generations.
Usage documentation can be found at ``cardano-cli/README.md``.
Expand All @@ -172,6 +171,9 @@ The general synopsis is as follows:
> NOTE: the exact invocation command depends on the environment. If you have only built ``cardano-cli``, without installing it, then you have to prepend ``cabal run -- ``
before ``cardano-cli``. We henceforth assume that the necessary environment-specific adjustment has been made, so we only mention ``cardano-cli``.

Command line options: ``cardano-cli``
====

The subcommands are subdivided in groups, and their full list can be seen in the output of ``cardano-cli --help``.

All subcommands have help available. For example:
Expand All @@ -194,11 +196,8 @@ All subcommands have help available. For example:
-h,--help Show this help text
Genesis operations
==================

Generation
----------
Genesis generation
====

The Byron genesis generation operations will create a directory that contains:

Expand Down Expand Up @@ -322,7 +321,7 @@ Local node queries

You can query the tip of your local node via the ``get-tip`` command as follows

1. Open `tmux`
1. Open ``tmux``
2. Run ``cabal build cardano-node``
3. Run ``./scripts/lite/shelley-testnet.sh example``
4. Run ``export CARDANO_NODE_SOCKET_PATH=/cardano-node/example/socket/node-1-socket
Expand Down Expand Up @@ -365,7 +364,7 @@ The mandatory arguments are ``--mainnet | --testnet-magic``, ``signing-key``, ``

The remaining arguments are optional parameters you want to update in your update proposal.

You can also check your proposal's validity using the `validate-cbor` command. See: `Validate CBOR files`_.
You can also check your proposal's validity using the ``validate-cbor`` command. See: `Validate CBOR files`_.

See the `Byron specification <https://hydra.iohk.io/job/Cardano/cardano-ledger-specs/byronLedgerSpec/latest/download-by-type/doc-pdf/ledger-spec>`_
for more details on update proposals.
Expand Down Expand Up @@ -413,54 +412,180 @@ Byron vote submission:
(--mainnet | --testnet-magic NATURAL)
--filepath UpdateProposalVoteFile
****
Development
===========
****

Overview
====

The ``cardano-node`` development is primarily based on the Nix infrastructure (https://nixos.org/ ), which enables packaging, CI, development environments and deployments.

On how to set up Nix for ``cardano-node`` development, please see `Building Cardano Node with nix <https://github.com/input-output-hk/cardano-node/tree/master/doc/getting-started/building-the-node-using-nix.md>`_.

Workbench: a local cluster playground
====

You can quickly spin up a local cluster (on Linux and Darwin), based on any of a wide variety of configurations, and put it under a transaction generation workload -- using the ``workbench`` environment:

1. Optional: choose a workbench profile:
- ``default`` stands for a light-state, 6-node cluster, under saturation workload, indefinite runtime
- ``ci-test`` is the profile run in the node CI -- very light, just two nodes and short runtime
- ``devops`` is an unloaded profile (no transaction generation) with short slots -- ``0.2`` sec.
- ..and many more -- which can be either:
- listed, by ``make ps``
- observed at their point of definition: `nix/workbench/profiles/prof1-variants.jq <https://github.com/input-output-hk/cardano-node/tree/master/nix/workbench/profiles/prof1-variants.jq#L333-L526>`_
2. Optional: select mode of operation, by optionally providing a suffix:
- default -- no suffix -- just enter the workbench shell, allowing you to run ``start-cluster`` at any time. Binaries will be built locally, by ``cabal``.
- ``autostay`` suffix -- enter the workbench shell, start the cluster, and stay in the shell afterwards. Binaries will be built locally, by ``cabal``.
- ``autonix`` suffix -- enter the workbench shell, start the cluster. All binaries will be provided by the Nix CI.
- ..there are other modes, as per `lib.mk <https://github.com/input-output-hk/cardano-node/tree/master/lib.mk>`_
3. Enter the workbench shell for the chosen profile & mode:
``make <PROFILE-NAME>`` or ``make <PROFILE-NAME>-<SUFFIX>`` (when there is a suffix).
4. Optional: start cluster:
Depending on the chosen mode, your cluster might already start, or you are expected to start it yourself, using ``start-cluster``.

The workbench services are available only inside the workbench shell.

Using Cabal
----

By default, all binaries originating in the ``cardano-node`` repository are available to ``cabal build`` and ``cabal run``, unless the workbench was entered using one of the pure ``*nix`` modes. Note that in all cases, the dependencies for the workbench are supplied though Nix and have been built/tested on CI.

**Dependency localisation** -or- *Cabal&Nix for painless cross-repository work*
----

The Cabal workflow described above only extends to the repository-local packages. Therefore, ordinarily, to work on ``cardano-node`` dependencies in the context of the node itself, one needs to go through an expensive multi-step process -- with committing, pushing and re-pinning of the dependency changes.

The **dependency localisation** workflow allows us to pick a subset of leaf dependencies of the ``cardano-node`` repository, and declare them *local* -- so they can be iterated upon using the ``cabal build`` / ``cabal run`` of ``cardano-node`` itself. This cuts development iteration time dramatically and enables effective cross-repo development of the full stack of Cardano packages.

Without further ado (**NOTE**: *the order of steps is important!*):

1. Ensure that your ``cardano-node`` checkout is clean, with no local modifications. Also, ensure that you start outside the node's Nix shell.
2. Check out the repository with the dependencies, *beside* the ``cardano-node`` checkout. You have to check out the git revision of the dependency used by your ``cardano-node`` checkout -- as listed in ``cardano-node/cabal.project``.
- we'll assume the ``ouroboros-network`` repository
- ..so a certain parent directory will include checkouts of both ``ouroboros-network`` and ``cardano-node``, at the same level
- ..and the git revision checked out in ``ouroboros-network`` will match the corresponding ``source-repository-package`` clause in ``cardano-node/cabal.project``.
- Extra point #1: you can localise/check out several dependency repositories
- Extra point #2: for the dependencies that are not listed in ``cabal.project`` of the node -- how do you determine the version to check out? You can ask the workbench shell:
1. Temporarily enter the workbench shell
2. Look for the package version in ``ghc-pkg list``
3. Use that version to determine the git revision of the dependency's repository (using a tag or some special knowledge about the version management of said dependency).
3. Enter the workbench shell, as per instructions in previous sections -- or just a plain Nix shell.
4. Ensure you can build ``cardano-node`` with Cabal: ``cabal build exe:cardano-node``. If you can't something else is wrong.
5. Determine the *leaf dependency set* you will have to work on. The *leaf dependency set* is defined to include the target package you want to modify, and its reverse dependencies -- that is, packages that depend on it (inside the dependency repository).
- let's assume, for example, that you want to modify ``ouroboros-consensus-shelley``
- ``ouroboros-consensus-shelley`` is not a leaf dependency in itself, since ``ouroboros-consensus-cardano`` (of the same ``ouroboros-network`` repository) depends on it -- so the *leaf dependency set* will include both of them.
- you might find out that you have to include a significant fraction of packages in ``ouroboros-network`` into this *leaf dependency set* -- do not despair.
- if the *leaf dependency set* is hard to determine, you can use ``cabal-plan`` -- which is included in the workbench shell (which you, therefore, have to enter temporarily):
.. code-block:: console
[nix-shell:~/cardano-node]$ cabal-plan dot-png --revdep ouroboros-consensus-shelley
This command will produce a HUGE ``deps.png`` file, which will contain the entire chart of the project dependencies. The important part to look for will be the subset of packages highlighted in red -- those, which belong to the ``ouroboros-network`` repository. This will be the full *leaf dependency set*.
6. Edit the ``cardano-node/cabal.project`` as follows:
- for the *leaf dependency set*
1. in the very beginning of the ``cabal.project``, add their relative paths to the ``packages:`` section, e.g.:
.. code-block:: console
packages:
cardano-api
cardano-cli
...
trace-resources
trace-forward
../ouroboros-network/ouroboros-consensus-shelley
../ouroboros-network/ouroboros-consensus-cardano
2. in the corresponding ``source-repository-package`` section, comment out mentions of the packages, e.g.:
.. code-block:: console
source-repository-package
type: git
location: https://github.com/input-output-hk/ouroboros-network
tag: c764553561bed8978d2c6753d1608dc65449617a
--sha256: 0hdh7xdrvxw943r6qr0xr4kwszindh5mnsn1lww6qdnxnmn7wcsc
subdir:
monoidal-synchronisation
network-mux
ouroboros-consensus
ouroboros-consensus-byron
-- ouroboros-consensus-cardano
ouroboros-consensus-protocol
-- ouroboros-consensus-shelley
ouroboros-network
ouroboros-network-framework
ouroboros-network-testing
7. The two packages have now became **local** -- when you try ``cabal build exe:cardano-node`` now, you'll see that Cabal starts to build these dependencies you just localised. Hacking time!

Hoogle
----

The workbench shell provides ``hoogle``, with a local database for the full set of dependencies:

.. code-block:: console
[nix-shell:~/cardano-node]$ hoogle search TxId
Byron.Spec.Ledger.UTxO newtype TxId
Byron.Spec.Ledger.UTxO TxId :: Hash -> TxId
Cardano.Chain.UTxO type TxId = Hash Tx
Cardano.Ledger.TxIn newtype TxId crypto
Cardano.Ledger.TxIn TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto
Cardano.Ledger.Shelley.API.Types newtype TxId crypto
Cardano.Ledger.Shelley.API.Types TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto
Cardano.Ledger.Shelley.Tx newtype TxId crypto
Cardano.Ledger.Shelley.Tx TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto
Ouroboros.Consensus.HardFork.Combinator data family TxId tx :: Type
-- plus more results not shown, pass --count=20 to see more
Supplementary tooling
====

GHCID
-----
----

run *ghcid* with: ``ghcid -c "cabal repl exe:cardano-node --reorder-goals"``

Haskell Language Server
-----------------------
----

When using Haskell Language Server with Visual Studio Code, you may find that
`HLINT annotations are ignored<https://github.com/haskell/haskell-language-server/issues/638>`.
`HLINT annotations are ignored <https://github.com/haskell/haskell-language-server/issues/638>`_.

To work around this, you may run the script `./scripts/reconfigure-hlint.sh` to generate a `.hlint.yaml`
To work around this, you may run the script ``./scripts/reconfigure-hlint.sh`` to generate a ``.hlint.yaml``
file with HLINT ignore rules derived from the source code.

Testing
========
====

``cardano-node`` is essentially a container which implements several components such networking, consensus, and storage. These components have individual test coverage. The node goes through integration and release testing by Devops/QA while automated CLI tests are ongoing alongside development.

Developers on ``cardano-node`` can `launch their own testnets <doc/getting-started/launching-a-testnet.md>`_ or `run the chairman tests <doc/getting-started/running-chairman-tests.md>`_ locally.

Chairman tests
--------------

Debugging
=========
====

Pretty printing CBOR encoded files
----------------------------------
----

It may be useful to print the on chain representations of blocks, delegation certificates, txs and update proposals. There are two commands that do this (for any cbor encoded file):

To pretty print as CBOR:
``cabal exec cardano-cli -- pretty-print-cbor --filepath CBOREncodedFile``

Validate CBOR files
-------------------
----

You can validate Byron era blocks, delegation certificates, txs and update proposals with the ``validate-cbor`` command.

``cabal exec cardano-cli -- validate-cbor --byron-block 21600 --filepath CBOREncodedByronBlockFile``


****
Native Tokens
=======================================
****

Native tokens is a new feature that enables the transacting of multi-assets on Cardano. Native tokens are now supported on mainnet and users can transact with ada, and an unlimited number of user-defined (custom) tokens natively.

Expand All @@ -472,10 +597,12 @@ To help you get started we have compiled a handy list of resources:

You can also read more about `native tokens and how they compare to ada and ERC20 <https://github.com/input-output-hk/cardano-ledger-specs/blob/master/doc/explanations/features.rst>`_. Browse native tokens created on the Cardano blockchain and see their transactions in an interactive dashboard that allows filtering and searching: nativetokens.da.iogservices.io.

****
API Documentation
=================
****

The API documentation is published `here <https://input-output-hk.github.io/cardano-node/>`_.

The documentation is built with each push, but is only published from `master` branch. In order to
test if the documentation is working, build the documentation locally with `./scripts/haddocs.sh` and
open `haddocks/index.html` in the browser.
The documentation is built with each push, but is only published from ``master`` branch. In order to
test if the documentation is working, build the documentation locally with ``./scripts/haddocs.sh`` and
open ``haddocks/index.html`` in the browser.
2 changes: 2 additions & 0 deletions nix/workbench/shell.nix
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ in project.shellFor {
ghcid
haskellBuildUtils
pkgs.graphviz
graphmod
cabal-plan
weeder
nixWrapped
pkgconfig
Expand Down
2 changes: 1 addition & 1 deletion trace-resources/src/Cardano/Logging/Resources/Darwin.hsc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ readResourceStatsInternal = getProcessID >>= \pid -> do
nsToCenti :: GhcStats.RtsTime -> Word64
nsToCenti = fromIntegral . (`div` 10000000)
timeValToCenti :: TIME_VALUE_T -> Word64
timeValToCenti tv = 10000 `div` (usFromTimeValue tv)
timeValToCenti tv = usFromTimeValue tv `div` 10000

usFromTimeValue :: TIME_VALUE_T -> Word64
usFromTimeValue (TIME_VALUE_T s us) = s * 1000000 + us

0 comments on commit e36c1a7

Please sign in to comment.