Store delegation reward account balances in the database #1768

rvl · 2020-06-16T14:06:36Z

Issue Number

Overview

Turns out querying reward account balances can be slow.
This is not good if it's done when listing/getting wallets.
Better to do these queries in a separate thread and cache the results in the database.

Add new database table for the latest known wallet reward account balance.
Use the database when getting the reward account balance.
Add a network layer method to watch the node tip.
Update the account balance in each wallet when the node tip changes (asynchronously).
Store reward account balance in db as it arrives.
Adapt Jörmungandr network layer

lib/core/src/Cardano/Wallet.hs

lib/core/src/Cardano/Wallet/Api/Server.hs

lib/core/src/Cardano/Wallet/DB.hs

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

lib/shelley/src/Cardano/Wallet/Shelley/Network.hs

KtorZ · 2020-06-23T09:45:26Z

lib/shelley/src/Cardano/Wallet/Shelley/Network.hs

+
+setupWatchers
+    :: Tracer IO (WatcherLog e)
+    -> TQueue IO v


There's already a lof of concurrent mutable variables down there, since the function only needs an accessor on the tip, I'd actually change this for just a IO v and not even mention the TQueue here.

OK good idea

lib/shelley/src/Cardano/Wallet/Shelley/Network.hs

KtorZ · 2020-06-27T16:25:38Z

bors r+

1768: Store delegation reward account balances in the database r=KtorZ a=rvl ### Issue Number ADP-302 / #1750 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer ### Comments - [ ] STAKE_POOLS_JOIN_04 test passes Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-06-27T16:42:12Z

Build failed

ci/hydra-eval

KtorZ · 2020-06-27T17:17:46Z

bors retry

1768: Store delegation reward account balances in the database r=KtorZ a=rvl ### Issue Number ADP-302 / #1750 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer ### Comments - [ ] STAKE_POOLS_JOIN_04 test passes Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-06-27T18:09:51Z

Canceled

KtorZ · 2020-06-27T19:11:12Z

bors r+

1768: Store delegation reward account balances in the database r=KtorZ a=rvl ### Issue Number ADP-302 / #1750 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer ### Comments - [ ] STAKE_POOLS_JOIN_04 test passes Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]> Co-authored-by: IOHK <[email protected]>

iohk-bors · 2020-06-27T19:33:40Z

Build failed

buildkite/cardano-wallet

KtorZ · 2020-06-27T19:40:09Z

bors retry

1768: Store delegation reward account balances in the database r=KtorZ a=rvl ### Issue Number ADP-302 / #1750 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer ### Comments - [ ] STAKE_POOLS_JOIN_04 test passes Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]> Co-authored-by: IOHK <[email protected]>

iohk-bors · 2020-06-27T20:03:52Z

Build failed

buildkite/cardano-wallet

iohk-bors · 2020-06-29T13:22:01Z

try

Build failed

buildkite/cardano-wallet

rvl · 2020-06-29T14:46:00Z

bors r+

1768: Store delegation reward account balances in the database r=rvl a=rvl ### Issue Number ADP-302 / #1750 / #1759 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]> Co-authored-by: Johannes Lund <[email protected]>

These wallet can be either Byron or Shelley wallets, so they don't necessarily have rewards capability.

KtorZ · 2020-06-29T14:52:33Z

bors r-

iohk-bors · 2020-06-29T14:52:36Z

Canceled

KtorZ · 2020-06-29T14:52:40Z

(just have some little changes to push)

KtorZ · 2020-06-29T14:53:54Z

bors r+

1768: Store delegation reward account balances in the database r=KtorZ a=rvl ### Issue Number ADP-302 / #1750 / #1759 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]> Co-authored-by: Johannes Lund <[email protected]>

Our intuition may be wrong here, and we need to investigate this.

iohk-bors · 2020-06-29T15:18:12Z

Canceled

KtorZ · 2020-06-29T15:23:21Z

bors r+

1768: Store delegation reward account balances in the database r=KtorZ a=rvl ### Issue Number ADP-302 / #1750 / #1759 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]> Co-authored-by: Johannes Lund <[email protected]>

iohk-bors · 2020-06-29T15:51:50Z

Build failed

buildkite/cardano-wallet

KtorZ · 2020-06-29T16:01:25Z

bors retry

1768: Store delegation reward account balances in the database r=KtorZ a=rvl ### Issue Number ADP-302 / #1750 / #1759 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]> Co-authored-by: Johannes Lund <[email protected]>

iohk-bors · 2020-06-29T16:21:30Z

Build failed

ci/hydra-build:required

rvl · 2020-06-30T00:30:55Z

Integration test failure seems to be due to a rolled back transaction ⇒ ADP-93. Retrying.

bors r+

1768: Store delegation reward account balances in the database r=rvl a=rvl ### Issue Number ADP-302 / #1750 / #1759 ### Overview Turns out querying reward account balances can be slow. This is not good if it's done when listing/getting wallets. Better to do these queries in a separate thread and cache the results in the database. - [x] Add new database table for the latest known wallet reward account balance. - [x] Use the database when getting the reward account balance. - [x] Add a network layer method to watch the node tip. - [x] Update the account balance in each wallet when the node tip changes (asynchronously). - [x] Store reward account balance in db as it arrives. - [x] Adapt Jörmungandr network layer Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: KtorZ <[email protected]> Co-authored-by: Johannes Lund <[email protected]>

iohk-bors · 2020-06-30T00:57:53Z

Build failed

buildkite/cardano-wallet

KtorZ · 2020-06-30T09:45:51Z

Merging, the failure above seems related to something else: #1831

rvl added this to the (ADP-302) Check reward account balance milestone Jun 16, 2020

rvl self-assigned this Jun 16, 2020

rvl force-pushed the rvl/adp-302/db-reward-balance branch from 5e078a6 to fa9ff16 Compare June 18, 2020 05:10

rvl mentioned this pull request Jun 18, 2020

Add logging and tracing to shelley integration tests #1754

Merged

4 tasks

KtorZ added the RESOLVING ISSUE Mark a PR as resolving issues, for auto-generated CHANGELOG label Jun 18, 2020

rvl force-pushed the rvl/adp-302/db-reward-balance branch 2 times, most recently from 40b5f06 to 6483eb6 Compare June 23, 2020 06:24

KtorZ approved these changes Jun 23, 2020

View reviewed changes

rvl force-pushed the rvl/adp-302/db-reward-balance branch 5 times, most recently from 78f5150 to 8574e98 Compare June 26, 2020 15:11

KtorZ force-pushed the rvl/adp-302/db-reward-balance branch 2 times, most recently from 90c5dba to f42de76 Compare June 27, 2020 12:14

KtorZ marked this pull request as ready for review June 27, 2020 13:07

KtorZ force-pushed the rvl/adp-302/db-reward-balance branch from 7184497 to 5102817 Compare June 27, 2020 16:24

KtorZ force-pushed the rvl/adp-302/db-reward-balance branch from 5102817 to caf9b46 Compare June 27, 2020 18:09

rvl and others added 5 commits June 29, 2020 16:49

Fix worker thread exiting on non-shelley wallets

c3bb4e8

Update assertion in NETWORK_PARAMS test

f9025e9

Update haddock in Registry

84ff792

Enable parts of "Rewards accumulate" test

3e504fb

pass coworker as argument for wallets creating from pub keys

007c9a6

These wallet can be either Byron or Shelley wallets, so they don't necessarily have rewards capability.

More comments about watchNodeTip in byron and jormungandr

c869327

KtorZ force-pushed the rvl/adp-302/db-reward-balance branch from 7532fab to c869327 Compare June 29, 2020 14:53

KtorZ added 2 commits June 29, 2020 17:10

disable integration tests which seems to make a false assumption

7cbbe17

Our intuition may be wrong here, and we need to investigate this.

turn off assertions which require emptying a reward account.

0a43be6

KtorZ merged commit bfc9408 into master Jun 30, 2020

KtorZ deleted the rvl/adp-302/db-reward-balance branch June 30, 2020 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store delegation reward account balances in the database #1768

Store delegation reward account balances in the database #1768

rvl commented Jun 16, 2020 •

edited

Loading

KtorZ Jun 23, 2020

rvl Jun 23, 2020

KtorZ commented Jun 27, 2020

iohk-bors bot commented Jun 27, 2020

KtorZ commented Jun 27, 2020

iohk-bors bot commented Jun 27, 2020

KtorZ commented Jun 27, 2020

iohk-bors bot commented Jun 27, 2020

KtorZ commented Jun 27, 2020

iohk-bors bot commented Jun 27, 2020

iohk-bors bot commented Jun 29, 2020

rvl commented Jun 29, 2020

KtorZ commented Jun 29, 2020

iohk-bors bot commented Jun 29, 2020

KtorZ commented Jun 29, 2020

KtorZ commented Jun 29, 2020

iohk-bors bot commented Jun 29, 2020

KtorZ commented Jun 29, 2020

iohk-bors bot commented Jun 29, 2020

KtorZ commented Jun 29, 2020

iohk-bors bot commented Jun 29, 2020

rvl commented Jun 30, 2020

iohk-bors bot commented Jun 30, 2020

KtorZ commented Jun 30, 2020

Store delegation reward account balances in the database #1768

Store delegation reward account balances in the database #1768

Conversation

rvl commented Jun 16, 2020 • edited Loading

Issue Number

Overview

KtorZ Jun 23, 2020

Choose a reason for hiding this comment

rvl Jun 23, 2020

Choose a reason for hiding this comment

KtorZ commented Jun 27, 2020

iohk-bors bot commented Jun 27, 2020

Build failed

KtorZ commented Jun 27, 2020

iohk-bors bot commented Jun 27, 2020

Canceled

KtorZ commented Jun 27, 2020

iohk-bors bot commented Jun 27, 2020

Build failed

KtorZ commented Jun 27, 2020

iohk-bors bot commented Jun 27, 2020

Build failed

iohk-bors bot commented Jun 29, 2020

try

Build failed

rvl commented Jun 29, 2020

KtorZ commented Jun 29, 2020

iohk-bors bot commented Jun 29, 2020

Canceled

KtorZ commented Jun 29, 2020

KtorZ commented Jun 29, 2020

iohk-bors bot commented Jun 29, 2020

Canceled

KtorZ commented Jun 29, 2020

iohk-bors bot commented Jun 29, 2020

Build failed

KtorZ commented Jun 29, 2020

iohk-bors bot commented Jun 29, 2020

Build failed

rvl commented Jun 30, 2020

iohk-bors bot commented Jun 30, 2020

Build failed

KtorZ commented Jun 30, 2020

rvl commented Jun 16, 2020 •

edited

Loading