PLT-359 ADR regarding problem with PAB's integration with indexing so…

…lutions

doc/adr/0001-record-architecture-decisions.rst

@@ -83,6 +83,13 @@ What follows is the ADR format (adapted from the book).
 |                      | For example, "ADR 1: Deployment on Ruby on Rails 3.0.10"                  |
 |                      | or "ADR 9: LDAP for Multitenant Integration"                              |
 +----------------------+---------------------------------------------------------------------------+
+| Status               | State the status of the decision, such as "draft" if the decision is      |
+|                      | still being written, as "proposed" if the project stakeholders haven't    |
+|                      | agreed with it yet, "accepted" once it is agreed. If a later ADR changes  |
+|                      | or reverses a decision, it may be marked as "deprecated" or "superseded"  |
+|                      | with a reference to its replacement. (This is not the status of           |
+|                      | implementing the decision.)                                               |
++----------------------+---------------------------------------------------------------------------+
 | Issue (or context)   | This section describes the architectural design issue being addressed.    |
 |                      | This description should leave no questions as to why this issue needs to  |
 |                      | be addressed now. The language in this section is value-neutral. It is    |
@@ -92,13 +99,6 @@ What follows is the ADR format (adapted from the book).
 |                      | positions that the architect could have taken. It is stated in full       |
 |                      | sentences, with active voice. "We will …"                                 |
 +----------------------+---------------------------------------------------------------------------+
-| Status               | State the status of the decision, such as "draft" if the decision is      |
-|                      | still being written, as "proposed" if the project stakeholders haven't    |
-|                      | agreed with it yet, "accepted" once it is agreed. If a later ADR changes  |
-|                      | or reverses a decision, it may be marked as "deprecated" or "superseded"  |
-|                      | with a reference to its replacement. (This is not the status of           |
-|                      | implementing the decision.)                                               |
-+----------------------+---------------------------------------------------------------------------+
 | Tags                 | Add one or more tags to the decision. Useful for organizing the set of    |
 |                      | decision.                                                                 |
 +----------------------+---------------------------------------------------------------------------+

doc/adr/0002-pab-indexing-solution-integration.rst

@@ -0,0 +1,142 @@
+ADR 2: PAB and indexing solution integration
+============================================
+
+Date: 2022-06-28
+
+Author(s)
+---------
+
+koslambrou <[email protected]>
+
+Status
+------
+
+Draft
+
+Context
+-------
+
+Let's start with the problematic example (copy-paste of the current `PubKey` contract in `plutus-use-cases`).
+
+.. code-block:: haskell
+
+  -- | Lock some funds in a 'PayToPubKey' contract, returning the output's address
+  --   and a 'TxIn' transaction input that can spend it.
+  pubKeyContract
+      :: forall w s e.
+      ( AsPubKeyError e
+      )
+      => PaymentPubKeyHash
+      -> Value
+      -> Contract w s e (TxOutRef, Maybe ChainIndexTxOut, TypedValidator PubKeyContract)
+  pubKeyContract pk vl = mapError (review _PubKeyError   ) $ do
+      -- Step 1
+      let inst = typedValidator pk
+          address = Scripts.validatorAddress inst
+          tx = Constraints.mustPayToTheScript () vl
+      ledgerTx <- mkTxConstraints (Constraints.typedValidatorLookups inst) tx
+                 >>= submitUnbalancedTx . Constraints.adjustUnbalancedTx
+
+      -- Step 2
+      _ <- awaitTxConfirmed (getCardanoTxId ledgerTx)
+
+      -- Step 3
+      let refs = Map.keys
+                 $ Map.filter ((==) address . txOutAddress)
+                 $ getCardanoTxUnspentOutputsTx ledgerTx
+      case refs of
+          []                   -> throwing _ScriptOutputMissing pk
+          [outRef] -> do
+              -- Step 4
+              ciTxOut <- unspentTxOutFromRef outRef
+              pure (outRef, ciTxOut, inst)
+          _                    -> throwing _MultipleScriptOutputs pk
+
+
+Here's an outline of the contract's steps:
+
+1. Creates a transaction and submits it to the node
+2. Waits for transaction to be confirmed
+3. Finds the first UTXO of that transaction (return type `TxOutRef`)
+4. Queries the plutus-chain-index to get the `ChainIndexTxOut` out of that `TxOutRef`
+
+The problem is that the `ciTxOut` variable in step 4 will almost always result in `Nothing`.
+
+Why? Here’s some context.
+
+The PAB listens to the local node and stores blockchain information in memory such as the status of transactions, the status of transaction outputs, the last synced slot, the current slot, etc., in a variable of type `BlockchainEnv`. The `awaitTxConfirmed` is actually querying the state of `BlockchainEnv` and waits until the status of the transaction transitions to `Confirmed`.
+
+Meanwhile, plutus-chain-index (our main indexing solution at the time of this writing) is also listening to incoming blocks from the local node and indexes them into a database. The indexed data can be queried using the REST API interface.
+
+This brings up the main issue: the PAB and plutus-chain-index each listen to the same source of information (a local Cardano node), but each index the information at different speeds. For a dApp developer writing off-chain code using the Contract API, there is no abstraction for handling multiple sources of truth.
+
+Currently, in the best case scenario (fully synced PAB and plutus-chain-index), plutus-chain-index will always trail behind the in-memory storage of the PAB by a few seconds. Therefore, even in this scenario, querying the plutus-chain-index with `unspentTxOutFromRef` in the above contract has a high probability of returning `Nothing`.
+
+Decision
+--------
+
+TBD
+
+Alternatives
+------------
+
+Make sure all components are in sync
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We can make sure that all indexing solutions are syncing at the same speed. For example, if PAB syncs from the local node and arrives at slot 100, then it needs to wait for the chain-index to also arrive at slot 100. Only then can it respond to a Contract request.
+
+To be able to do that, we would need to modify the block listener of PAB and plutus-chain-index to only get new blocks once they are at the same slot.
+
+Pros:
+
+- No modification to the Contract API
+- All the indexed information is consistent with each other
+
+Cons:
+
+- Hard to implement
+- As fast as the slowest indexing solution
+- Can not be feasible with external indexing solutions for which we can’t control the indexing speed (e.g. Blockfrost_ or Scrolls_)
+
+Add indexing specific functions in the Contract API
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this scenario, we would need to split Contract API requests which interact with an external indexing solution to the ones that use the PAB. Currently, we have `awaitTxConfirmed` which uses the indexed information in the PAB to wait for a transaction status to change to `Confirmed`. On top of that, we can have `awaitTxIndexed` or `awaitTxOutIndexed` which will wait for the information to be indexed in the external indexing solution.
+
+Pros:
+
+- Limits design change on the PAB
+- More control given to the user of the Contract API
+
+Cons:
+
+- Adds (undesirable?) complexity to the Contract API
+- We'll need to add a bunch of functions (e.g., `awaitMarconiTxConfirmed or `awaitScrollsTxConfirmed`) for each new indexing solution we want to support
+
+Query functions should interact with a single source of truth
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this scenario, we make the design decision that the Contract API should only interact with a single indexing solution. Thus, any blockchain information currently stored in the PAB should be moved to the indexing solution. Also, combining indexing solutions would need to be integrated in the single indexing solution that’s connected to the PAB.
+
+Pros:
+
+- Simplest in design to implement (other than manual work to move code)
+- No modification to the Contract API
+
+Cons:
+
+- PAB won't be able to work with external indexing solutions (e.g. Blockfrost_ or Scrolls_)
+
+Implications
+------------
+
+TBD
+
+Notes
+-----
+
+This problem manifested itself in the Github issue `#473 <https://github.com/input-output-hk/plutus-apps/issues/473>`_ and there was a temporary fix in the PR `#496 <https://github.com/input-output-hk/plutus-apps/pull/496>`_.
+However, the proper solution to the issue would be the implementation of this ADR.
+
+.. _Blockfrost: https://blockfrost.io
+.. _Scrolls: https://github.com/txpipe/scrolls

doc/adr/index.rst

@@ -31,3 +31,4 @@ The general process for creating an ADR is:
    :titlesonly:
 
    0001-record-architecture-decisions
+   0002-pab-indexing-solution-integration