Merge #2123 #2138 #2161

2123: Less verbose integration test output r=KtorZ a=Anviking

# Issue Number

#2119 


# Overview

<!-- Detail in a few bullet points the work accomplished in this PR -->

- [x] More quiet stdout per default when running integration tests
- [x] Log full wallet, node and test-setup logs to `$TESTS_LOGDIR`, if set.
- [x] Upload `$TESTS_LOGDIR/*.log` as Buildkite artefacts

# Comments

- Regardless of `$TESTS_LOGDIR` setting, logs are still written to the temporary cluster directory as before.
- We can make hydra and GitHub Actions upload logs in a future PR. We might also be fine without it.

<img width="1458" alt="Skärmavbild 2020-09-10 kl  17 57 58" src="https://user-images.githubusercontent.com/304423/92758896-79597480-f38f-11ea-9394-501dc4608221.png">


<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


2138: Save intermediate state when restoring the Byron chain in the stake pools worker r=KtorZ a=hasufell

This completely skips byron era for mainnet when syncing stake pools.

I extracted the first shelley block by looking at a fully synced stake pool DB and selecting the lowest `block_height` from the `pool_production` table.

For non-mainnet it will record the last byron block of a chunk from the node into the `byron_headers` table and read it out during `readPoolProductionCursor` if there are no stake pools and otherwise ignore it.

Remarks:

- the "genesis shelley header" is hardcoded, because the node currently does not have an endpoint to get this

2161: cache latest checkpoint in-memory to avoid deserializing and reserializing it too often r=KtorZ a=KtorZ

# Issue Number

<!-- Put here a reference to the issue this PR relates to and which requirements it tackles -->


# Overview

<!-- Detail in a few bullet points the work accomplished in this PR -->



  This little change has some drastic performance improvements on
  various tasks of the wallet:

  ```
  Wallet Overview
     number of addresses: 29209
     number of transactions: 48819
     number of utxos: 2778 UTxOs

  Bench                   | No Cache | With Cache
  ---                     | --       | ---
  restoreTime             | 63min    | 22min
  readWalletTime          | 715ms    | 53.70 ms
  listAddressesTime       | 1.863s   | 2.169 ms
  listTransactionsTime    | 75.72s   | 12.68 s
  importOneAddressTime    | 2.526s   | 260.4 ms
  importManyAddressesTime | 3.104s   | 1.690 s
  estimateFeesTime        | 2.103s   | 63.11 ms
  ```

  This cache approach however supposes that there's only a single wallet
  per database, which is the case and has been the case for a long time
  now. I'll remove the WalletId from the database interface in a next
  commit to clean things up a bit.



# Comments

<!-- Additional comments or screenshots to attach if any -->

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: Johannes Lund <[email protected]>
Co-authored-by: Rodney Lorrimar <[email protected]>
Co-authored-by: KtorZ <[email protected]>
Co-authored-by: Julian Ospald <[email protected]>

.buildkite/pipeline.yml

@@ -3,6 +3,7 @@ env:
   STACK_ROOT: "/build/cardano-wallet.stack"
   CACHE_DIR: "/cache/cardano-wallet"
   LC_ALL: "en_US.UTF-8"
+  TESTS_LOGDIR: "/tmp/wallet-integration-logs"
 
 steps:
   - label: 'Prevent merging to wrong branch'
@@ -15,11 +16,13 @@ steps:
 
   - label: 'Stack Rebuild'
     command:
+      - "mkdir -p $TESTS_LOGDIR"
       - "nix-build .buildkite/default.nix -o sr"
       - "./sr/bin/rebuild --build-dir=$BUILD_DIR --cache-dir=$CACHE_DIR"
     timeout_in_minutes: 120
     artifact_paths:
       - "/build/cardano-wallet/.stack-work/logs/cardano-wallet*.log"
+      - "/tmp/wallet-integration-logs/*.log"
     agents:
       system: x86_64-linux
 

lib/cli/src/Cardano/CLI.hs

@@ -58,6 +58,7 @@ module Cardano.CLI
 
     -- * Option parsers for configuring tracing
     , LoggingOptions (..)
+    , LogOutput (..)
     , helperTracing
     , loggingOptions
     , loggingSeverities
@@ -100,6 +101,13 @@ import Cardano.BM.Backend.Switchboard
     ( Switchboard )
 import Cardano.BM.Configuration.Static
     ( defaultConfigStdout )
+import Cardano.BM.Data.Output
+    ( ScribeDefinition (..)
+    , ScribeFormat (..)
+    , ScribeId
+    , ScribeKind (..)
+    , ScribePrivacy (..)
+    )
 import Cardano.BM.Data.Severity
     ( Severity (..) )
 import Cardano.BM.Data.Tracer
@@ -1488,34 +1496,61 @@ data Verbosity
         -- ^ Include more information in the log output.
     deriving (Eq, Show)
 
+data LogOutput
+    = LogToStdout Severity
+    | LogToFile FilePath Severity
+    deriving (Eq, Show)
+
+
+mkScribe :: LogOutput -> ScribeDefinition
+mkScribe (LogToStdout sev) = ScribeDefinition
+    { scName = "text"
+    , scFormat = ScText
+    , scKind = StdoutSK
+    , scMinSev = sev
+    , scMaxSev = Critical
+    , scPrivacy = ScPublic
+    , scRotation = Nothing
+    }
+mkScribe (LogToFile path sev) = ScribeDefinition
+    { scName = T.pack path
+    , scFormat = ScText
+    , scKind = FileSK
+    , scMinSev = sev
+    , scMaxSev = Critical
+    , scPrivacy = ScPublic
+    , scRotation = Nothing
+    }
+
+mkScribeId :: LogOutput -> ScribeId
+mkScribeId (LogToStdout _) = "StdoutSK::text"
+mkScribeId (LogToFile file _) = T.pack $ "FileSK::" <> file
+
+
 -- | Initialize logging at the specified minimum 'Severity' level.
 initTracer
-    :: Maybe FilePath
-    -> Severity
+    :: [LogOutput]
     -> IO (Switchboard Text, (CM.Configuration, Trace IO Text))
-initTracer configFile minSeverity = do
-    let defaultConfig = do
-            c <- defaultConfigStdout
-            CM.setMinSeverity c minSeverity
-            CM.setSetupBackends c [CM.KatipBK, CM.AggregationBK]
-            pure c
-    cfg <- maybe defaultConfig CM.setup configFile
+initTracer outputs = do
+    cfg <- do
+        c <- defaultConfigStdout
+        CM.setSetupBackends c [CM.KatipBK, CM.AggregationBK]
+        CM.setSetupScribes c $ map mkScribe outputs
+        CM.setDefaultScribes c $ map mkScribeId outputs
+        pure c
     (tr, sb) <- setupTrace_ cfg "cardano-wallet"
     pure (sb, (cfg, tr))
 
 -- | Run an action with logging available and configured. When the action is
 -- finished (normally or otherwise), log messages are flushed.
 withLogging
-    :: Maybe FilePath
-    -- ^ Configuration file - uses default otherwise.
-    -> Severity
-    -- ^ Minimum severity level to log
+    :: [LogOutput]
     -> ((CM.Configuration, Trace IO Text) -> IO a)
     -- ^ The action to run with logging configured.
     -> IO a
-withLogging configFile minSeverity action = bracket before after (action . snd)
+withLogging outputs action = bracket before after (action . snd)
   where
-    before = initTracer configFile minSeverity
+    before = initTracer outputs
     after (sb, (_, tr)) = do
         logDebug (appendName "main" tr) "Logging shutdown."
         shutdown sb