CAD-391 eliding messages #445

CodiePP · 2019-12-24T13:40:01Z

prints first and last message of range of equivalent 'TraceLedgerReplayEvent'.

[nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.00 UTC] {"event":{"kind":"TraceOpenEvent.OpenedVolDB"},"kind":"TraceEvent","tip":"genesis(origin)"}
[nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.31 UTC] {"event":{"snapshot":{"snapshot":"DiskSnapshot 9","kind":"snapshot"},"kind":"TraceLedgerReplayEvent.ReplayFromSnapshot","tip":"Tip (At (Block {blockPointSlot = SlotNo {unSlotNo = 3549092}, blockPointHash = ByronHash {unByronHash = AbstractHash 2cfa35fe9617a9c41ef5ec19bed50742e587ab10d18ce6ac80c09bf71308fc3e}}))"},"kind":"TraceEvent","tip":"genesis(origin)"}
[nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] {"event":{"snapshot":{"snapshot":"DiskSnapshot 9","kind":"snapshot"},"kind":"TraceLedgerReplayEvent.ReplayFromSnapshot","tip":"Tip (At (Block {blockPointSlot = SlotNo {unSlotNo = 3549092}, blockPointHash = ByronHash {unByronHash = AbstractHash 2cfa35fe9617a9c41ef5ec19bed50742e587ab10d18ce6ac80c09bf71308fc3e}}))"},"kind":"TraceEvent","tip":"genesis(origin)"}
[nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] messages elided in total = 70
[nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] {"event":{"kind":"TraceOpenEvent.OpenedLgrDB"},"kind":"TraceEvent","tip":"genesis(origin)"}

CodiePP · 2019-12-27T15:36:28Z

depends on PR input-output-hk/iohk-monitoring-framework#495

CodiePP · 2019-12-30T13:13:35Z

eliding TraceLedgerReplayEvent:

[nixos19:cardano.node.ChainDB:Info:63] [2019-12-30 13:06:34.52 UTC] {"event":{"snapshot":{"snapshot":"DiskSnapshot 19","kind":"snapshot"},"kind":
"TraceLedgerReplayEvent.ReplayFromSnapshot","tip":"Tip (At (Block {blockPointSlot = SlotNo {unSlotNo = 3560492}, blockPointHash = ByronHash {unBy
ronHash = AbstractHash 3ac09146dbc3c75e298543a22223ac3aeeab387d9ddecedac5cbe6465db34dc0}}))"},"kind":"TraceEvent","tip":"genesis(origin)"}
[nixos19:cardano.node.ChainDB:Info:63] [2019-12-30 13:06:34.78 UTC] before next, messages elided = 1432
[nixos19:cardano.node.ChainDB:Info:63] [2019-12-30 13:06:34.78 UTC] {"event":{"kind":"TraceLedgerReplayEvent.ReplayedBlock","slot":"3542400","epoch":"164"},"kind":"TraceEvent","tip":"genesis(origin)"}

eliding TraceGCEvent:

[nixos19:cardano.node.ChainDB:Debug:79] [2019-12-30 13:07:09.88 UTC] {"event":{"kind":"TraceGCEvent.PerformedGC","slot":{"kind":"SlotNo","slot":3562021}},"kind":"TraceEvent","tip":"4cac1a8@3567746"}
[nixos19:cardano.node.ChainDB:Debug:81] [2019-12-30 13:07:09.88 UTC] before next, messages elided = 2
[nixos19:cardano.node.ChainDB:Debug:81] [2019-12-30 13:07:09.88 UTC] {"event":{"kind":"TraceGCEvent.PerformedGC","slot":{"kind":"SlotNo","slot":3562025}},"kind":"TraceEvent","tip":"4cac1a8@3567746"}

CodiePP · 2019-12-30T17:47:28Z

depends on PR input-output-hk/iohk-monitoring-framework#495

upstream PR has been merged. now ready.

Signed-off-by: Alexander Diemand <[email protected]>

ksaric

Not sure about some of the stuff, left comments.

ksaric · 2020-01-07T13:56:24Z

benchmarking/chain-sync/benchmark-chain-sync-mainnet.sh

-  rm -rf db-mainnet-0
-fi
-rm node-0*
+#if [ -d db-mainnet-0 ]; then


Do you need to remove this? I'm not sure if this is intentional.

hm, no, this is from debugging

ksaric · 2020-01-07T13:57:19Z

benchmarking/chain-sync/benchmark-chain-sync-mainnet.sh


 date --iso-8601=seconds > STARTTIME

 NODE="cabal v2-run exe:cardano-node -- "
+NODE="stack --nix exec cardano-node -- "


This probably needs an if.

yes, switched to 'stack'. Changing the order of the definitions will use stack/cabal.

ksaric · 2020-01-07T13:57:55Z

benchmarking/chain-sync/configuration/log-configuration.yaml

@@ -1,5 +1,5 @@
 # global filter; messages must have at least this severity to pass:
-minSeverity: Notice
+minSeverity: Debug


Just the Debug messages?

ksaric · 2020-01-07T13:58:13Z

benchmarking/chain-sync/configuration/log-configuration.yaml

@@ -68,6 +68,8 @@ options:
      # Change the `subtrace` value to `Neutral` in order to log
      # `UTxO`-related messages during epoch validation.
      subtrace: NoTrace
+    'cardano.#messagecounters':


Do we even need this, since input-output-hk/iohk-monitoring-framework#496 ?

once that PR is merged, then no more.

ksaric · 2020-01-07T13:58:41Z

benchmarking/chain-sync/launch_mainnet_proxy.sh

@@ -11,7 +11,10 @@ set -euo pipefail

 cd ${TARGETDIR}

-cabal v2-run exe:cardano-byron-proxy -- \
+RUNNER="cabal v2-run exe:cardano-byron-proxy --"
+RUNNER="stack --nix exec cardano-byron-proxy --"


It probably needs an if as well.

ksaric · 2020-01-07T14:00:06Z

cardano-node/src/Cardano/Tracing/Tracers.hs

+instance ElidingTracer
+  (WithSeverity (WithTip blk (ChainDB.TraceEvent blk))) where
+  -- equivalent by type and severity
+  isEquivalent (WithSeverity s1 (WithTip _tip1 (ChainDB.TraceLedgerReplayEvent _ev1)))


If they are both TraceLedgerReplayEvent and have the same severity? No need to match on the actual event of TraceLedgerReplayEvent?

equivalence just by comparing severity and the type; could be made more complicated..

Signed-off-by: Alexander Diemand <[email protected]>

ksaric

LGTM.

CodiePP · 2020-01-08T09:45:00Z

bors r+

445: CAD-391 eliding messages r=CodiePP a=CodiePP prints first and last message of range of equivalent 'TraceLedgerReplayEvent'. ``` [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.00 UTC] {"event":{"kind":"TraceOpenEvent.OpenedVolDB"},"kind":"TraceEvent","tip":"genesis(origin)"} [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.31 UTC] {"event":{"snapshot":{"snapshot":"DiskSnapshot 9","kind":"snapshot"},"kind":"TraceLedgerReplayEvent.ReplayFromSnapshot","tip":"Tip (At (Block {blockPointSlot = SlotNo {unSlotNo = 3549092}, blockPointHash = ByronHash {unByronHash = AbstractHash 2cfa35fe9617a9c41ef5ec19bed50742e587ab10d18ce6ac80c09bf71308fc3e}}))"},"kind":"TraceEvent","tip":"genesis(origin)"} [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] {"event":{"snapshot":{"snapshot":"DiskSnapshot 9","kind":"snapshot"},"kind":"TraceLedgerReplayEvent.ReplayFromSnapshot","tip":"Tip (At (Block {blockPointSlot = SlotNo {unSlotNo = 3549092}, blockPointHash = ByronHash {unByronHash = AbstractHash 2cfa35fe9617a9c41ef5ec19bed50742e587ab10d18ce6ac80c09bf71308fc3e}}))"},"kind":"TraceEvent","tip":"genesis(origin)"} [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] messages elided in total = 70 [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] {"event":{"kind":"TraceOpenEvent.OpenedLgrDB"},"kind":"TraceEvent","tip":"genesis(origin)"} ``` Co-authored-by: Alexander Diemand <[email protected]>

iohk-bors · 2020-01-08T10:05:17Z

Build failed

buildkite/cardano-node

CodiePP · 2020-01-08T13:01:49Z

bors r+

445: CAD-391 eliding messages r=CodiePP a=CodiePP prints first and last message of range of equivalent 'TraceLedgerReplayEvent'. ``` [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.00 UTC] {"event":{"kind":"TraceOpenEvent.OpenedVolDB"},"kind":"TraceEvent","tip":"genesis(origin)"} [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.31 UTC] {"event":{"snapshot":{"snapshot":"DiskSnapshot 9","kind":"snapshot"},"kind":"TraceLedgerReplayEvent.ReplayFromSnapshot","tip":"Tip (At (Block {blockPointSlot = SlotNo {unSlotNo = 3549092}, blockPointHash = ByronHash {unByronHash = AbstractHash 2cfa35fe9617a9c41ef5ec19bed50742e587ab10d18ce6ac80c09bf71308fc3e}}))"},"kind":"TraceEvent","tip":"genesis(origin)"} [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] {"event":{"snapshot":{"snapshot":"DiskSnapshot 9","kind":"snapshot"},"kind":"TraceLedgerReplayEvent.ReplayFromSnapshot","tip":"Tip (At (Block {blockPointSlot = SlotNo {unSlotNo = 3549092}, blockPointHash = ByronHash {unByronHash = AbstractHash 2cfa35fe9617a9c41ef5ec19bed50742e587ab10d18ce6ac80c09bf71308fc3e}}))"},"kind":"TraceEvent","tip":"genesis(origin)"} [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] messages elided in total = 70 [nixos19:cardano.node.ChainDB:Info:64] [2019-12-24 23:26:57.39 UTC] {"event":{"kind":"TraceOpenEvent.OpenedLgrDB"},"kind":"TraceEvent","tip":"genesis(origin)"} ``` Co-authored-by: Alexander Diemand <[email protected]>

iohk-bors · 2020-01-08T13:16:56Z

Build succeeded

CodiePP added the WIP Work In Progress (cannot be merged yet) label Dec 24, 2019

CodiePP changed the title ~~CAD-391 WIP: experimenting with eliding messages~~ CAD-391 eliding messages Dec 24, 2019

CodiePP requested a review from mrBliss December 24, 2019 23:33

CodiePP removed the WIP Work In Progress (cannot be merged yet) label Dec 24, 2019

CodiePP added this to the S3 2020-01-02 milestone Dec 24, 2019

CodiePP marked this pull request as ready for review December 24, 2019 23:34

CodiePP mentioned this pull request Dec 24, 2019

[#303] more information on replay events #309

Closed

CodiePP added the WIP Work In Progress (cannot be merged yet) label Dec 27, 2019

CodiePP force-pushed the cad-391-log-startup branch from 459492e to 355f39d Compare December 30, 2019 12:06

CodiePP requested a review from Jimbo4350 December 30, 2019 13:05

CodiePP removed the WIP Work In Progress (cannot be merged yet) label Dec 30, 2019

CodiePP added 5 commits January 3, 2020 17:30

CAD-391 WIP: experimenting with eliding messages

cd10846

Signed-off-by: Alexander Diemand <[email protected]>

CAD-391 eliding equivalent messages

878726e

Signed-off-by: Alexander Diemand <[email protected]>

moved eliding tracer transformer to iohk-monitoring

0e3b3b2

Signed-off-by: Alexander Diemand <[email protected]>

CAD-391 nixified

3a8126b

Signed-off-by: Alexander Diemand <[email protected]>

CAD-391 output every 100th elided message

54c8875

CodiePP force-pushed the cad-391-log-startup branch from f19beac to 54c8875 Compare January 3, 2020 16:33

CodiePP modified the milestones: S3 2020-01-02, S4 2020-01-16 Jan 3, 2020

CodiePP requested review from deepfire and ksaric January 7, 2020 07:53

ksaric reviewed Jan 7, 2020

View reviewed changes

CAD-391 cleanup script

cdfe112

Signed-off-by: Alexander Diemand <[email protected]>

ksaric approved these changes Jan 8, 2020

View reviewed changes

iohk-bors bot merged commit cdfe112 into master Jan 8, 2020

iohk-bors bot deleted the cad-391-log-startup branch January 8, 2020 13:17

CodiePP mentioned this pull request Jan 10, 2020

CAD-391 Add some logging during startup #303

Closed

CodiePP added the byron Required for a Byron mainnet: replace the old core nodes with cardano-node. label Jan 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CAD-391 eliding messages #445

CAD-391 eliding messages #445

CodiePP commented Dec 24, 2019 •

edited

Loading

CodiePP commented Dec 27, 2019

CodiePP commented Dec 30, 2019

CodiePP commented Dec 30, 2019

ksaric left a comment

ksaric Jan 7, 2020

CodiePP Jan 7, 2020

ksaric Jan 7, 2020

CodiePP Jan 7, 2020

ksaric Jan 7, 2020

ksaric Jan 7, 2020

CodiePP Jan 7, 2020

ksaric Jan 7, 2020

ksaric Jan 7, 2020

CodiePP Jan 7, 2020

ksaric left a comment

CodiePP commented Jan 8, 2020

iohk-bors bot commented Jan 8, 2020

CodiePP commented Jan 8, 2020

iohk-bors bot commented Jan 8, 2020

CAD-391 eliding messages #445

CAD-391 eliding messages #445

Conversation

CodiePP commented Dec 24, 2019 • edited Loading

CodiePP commented Dec 27, 2019

CodiePP commented Dec 30, 2019

CodiePP commented Dec 30, 2019

ksaric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ksaric left a comment

Choose a reason for hiding this comment

CodiePP commented Jan 8, 2020

iohk-bors bot commented Jan 8, 2020

Build failed

CodiePP commented Jan 8, 2020

iohk-bors bot commented Jan 8, 2020

Build succeeded

CodiePP commented Dec 24, 2019 •

edited

Loading