[Epic] Wallet Service v1

[obiyankenobi]

Introduction

The hathor-wallet-service will be used as a backend for all official Hathor's wallets.

This service is highly inspired by the Bitcore Wallet Service.

Motivation

The idea behind this service is to have a fully indexed Hathor blockchain in real-time on a centralized database so we don't have to seek the full-node to calculate his balances and to query his utxos it every time a user wants to use his wallet, thereby making the app a lot faster for large wallets and reducing the load on the full nodes.

Acceptance criteria

Aside from faster load times, the apps that are using this service as their backends should have all features working exactly as they were working when connected directly on the fullnode.

We should have control over exactly what percentage of our user base should be using this service and these percentages should be configured for each platform individually

Q3 Week 7: Aug 16th - Aug 20th

• feat: match library facades signature methods (HathorWallet and HathorWalletServiceWallet) hathor-wallet-lib#215 - @pedroferreira1
• feat: wallet service integration hathor-wallet-mobile#101 - @pedroferreira1
• feat: wallet service integration with optimizely for internal testing hathor-wallet-mobile#114 - @andreabadesso
• feat: add real time update on the app when running the wallet service facade hathor-wallet-mobile#113 - @andreabadesso
• feat: the daemon should send the voided transactions when received from the fullnode hathor-wallet-service#8 - @andreabadesso

Q3 Week 6: Aug 9th - Aug 13th

• [Partial] feat: wallet service integration with optimizely for internal testing hathor-wallet-mobile#114 - @andreabadesso
  • Made some progress this week, still needs more work to complete
• [Fail] feat: add real time update on the app when running the wallet service facade hathor-wallet-mobile#113 - @andreabadesso
  • No progress this week
• [Fail] feat: the daemon should send the voided transactions when received from the fullnode hathor-wallet-service#8 - @andreabadesso
  • No progress this week
• [Partial] feat: match library facades signature methods (HathorWallet and HathorWalletServiceWallet) hathor-wallet-lib#215 - @pedroferreira1
• [Partial] feat: wallet service integration hathor-wallet-mobile#101 - @pedroferreira1

In Review (from previous weeks)

• [Done] feat: lavamoat on wallet-service hathor-wallet-service#112 - @andreabadesso
• [Done] chore(deps): bump json5, @size-limit/preset-small-lib and tsconfig-paths hathor-wallet-service#82 - @andreabadesso
• [Partial] fix: missing walletservice env hathor-wallet-service#117 - @andreabadesso
  • There is an ongoing discussion about how to wrap the method with a db transaction. I refactored it to the latest @msbrogli suggestion and now the issue is pending a re-review
• [Partial] fix: codebuild failing because of yarn version hathor-wallet-service#116 - @andreabadesso
  • This PR has 3 approves, but they're still one change that @msbrogli asked for. The change is done, pending a re-review on that specific point: fix: missing walletservice env hathor-wallet-service#117 (comment)

Q3 Week 5: Aug 2nd - Aug 6th

• [Done] feat: add support for new token actions in the wallet lib hathor-wallet-lib#197 - @pedroferreira1
• [Partial] feat: match library facades signature methods (HathorWallet and HathorWalletServiceWallet) hathor-wallet-lib#215 - @pedroferreira1
• [Partial] feat: wallet service integration hathor-wallet-mobile#101 - @pedroferreira1
• [Done] chore: updated package-lock.json to v2 hathor-wallet-service#75 - @andreabadesso
• [In Review] fix: codebuild failing because of yarn version hathor-wallet-service#116 - @andreabadesso
  • This issue replaced feat: handle mempool voided transactions on wallet-service #89 with a different approach. It is ready to review
• [Partial] feat: the daemon should send the voided transactions when received from the fullnode hathor-wallet-service#8 - @andreabadesso
  • Didn't work on this issue this week, still pending some tests, but implementation is ready
• [In Review] fix: broken deploy hathor-wallet-service#114 - @andreabadesso
  • Done and currently in review
• [Fail] refactor: fetch reward min blocks hathor-wallet-service#101 - @andreabadesso
  • No progress yet
• [Fail] feat: add real time update on the app when running the wallet service facade hathor-wallet-mobile#113 - @andreabadesso
  • No progress yet

Unplanned issues

• [Done] fix: send height on transactions hathor-wallet-service#18 - @andreabadesso

Q3 Week 4: Jul 26th - Jul 30th

• [Partial] feat: add support for new token actions in the wallet lib hathor-wallet-lib#197 - @pedroferreira1
• [Partial] feat: match library facades signature methods (HathorWallet and HathorWalletServiceWallet) hathor-wallet-lib#215 - @pedroferreira1
• [Partial] feat: wallet service integration hathor-wallet-mobile#101 - @pedroferreira1
• [Fail] Implement support for wallet service rollout hathor-wallet#208 - @pedroferreira1
• [Partial] feat: realiable integrations [part 1 - machine] hathor-wallet-service#89 - @andreabadesso
  • I spent very little time on this issue, still needs ~4h work to finish it
• [Partial] feat: the daemon should send the voided transactions when received from the fullnode hathor-wallet-service#8 - @andreabadesso
  • Also spent very little time on it, needs to improve tests to open the PR
• [Partial] refactor: fetch reward min blocks hathor-wallet-service#101 - @andreabadesso
  • Had some progress on it, identified some issues with the loadWallet, still some work to be done
• [In Review] feat: healthcheck API hathor-wallet-service#110 - @andreabadesso
  • Done and currently in review
• [Done] fix: invalid sqs message hathor-wallet-service#107 - @andreabadesso
  • Done in the beginning of the week
• [Fail] chore: test with large wallets (20k+ txs) - @andreabadesso
  • Didn't work on this issue
• [Moved to parking lot] fix: handling throwing lambda call hathor-wallet-service#84 - @r4mmer

Q3 Week 3: Jul 19th - Jul 23rd

• [Partial] feat: add support for new token actions in the wallet lib hathor-wallet-lib#197 - @pedroferreira1
• [Partial] feat: match library facades signature methods (HathorWallet and HathorWalletServiceWallet) hathor-wallet-lib#215 - @pedroferreira1
• [Partial] feat: wallet service integration hathor-wallet-mobile#101 - @pedroferreira1
• [Fail] Implement support for wallet service rollout hathor-wallet#208 - @pedroferreira1
• [Partial] feat: realiable integrations [part 1 - machine] hathor-wallet-service#89 - @andreabadesso
  • Had some progress on this PR, I have a working proof of concept developed but decided on a new, simpler mechanism for voiding transactions that is yet to be discussed and developed.
• [Partial] feat: the daemon should send the voided transactions when received from the fullnode hathor-wallet-service#8 - @andreabadesso
  • Also had some progress last week, I am sending the voided transactions as they are received from the WebSocket but depending on feat: handle mempool voided transactions on wallet-service #89, there will be some changes necessary on the daemon as well
• [Partial] fix: invalid sqs message hathor-wallet-service#107 - @andreabadesso
  • This is already approved by @msbrogli and @pedroferreira1 but is still pending an answer from @msbrogli so it can be merged
• [Partial] refactor: fetch reward min blocks hathor-wallet-service#101 - @andreabadesso
  • Listed some queries that need to be worked on and opened a PR (refactor: getLatestHeight now uses LIMIT 1 to get the latest block #108) that should've been part of this issue but had to be merged so the daemon wouldn't stop every time this function was called.
• [Fail] chore: test with large wallets (20k+ txs) - @andreabadesso
  • No progress
• [Fail] Discuss with @msbrogli the feasibility of using the fullnode database directly instead of the current sync implementation
  • Did not discuss
• [Partial] fix: handling throwing lambda call hathor-wallet-service#84 - @r4mmer
  • Implementation is done and tested, but there are still some pending discussions about timeout configuration and potential optimizations to the auth implementation
• [Partial] chore(deps): bump decode-uri-component from 0.2.0 to 0.2.2 hathor-wallet-service#80 - @luislhl
  • @luislhl finished the design with suggestions for each point, those suggestions are being discussed on the PR

Unplanned issues:

• [Done] fix: push aws region hathor-wallet-service#108 - @andreabadesso
• [Done] feat: websocket conn error log hathor-wallet-service#17 - @andreabadesso
• [Done] feat: emitting connection_error if a connection fails hathor-wallet-lib#216 - @andreabadesso

Q3 Week 2: Jul 12th - Jul 16th

• [Partial] feat: add support for new token actions in the wallet lib hathor-wallet-lib#197 - @pedroferreira1
• [Partial] feat: wallet service integration hathor-wallet-mobile#101 - @pedroferreira1
• [Partial] Implement support for wallet service rollout hathor-wallet#208 - @pedroferreira1
• [In Review] chore(deps): bump json5, @size-limit/preset-small-lib and tsconfig-paths hathor-wallet-service#82 - @andreabadesso
  • The txProposalSend API was refactored entirely. This is currently deployed on a Lambda staging environment and was already tested by @pedroferreira1
• [In Review] chore: updated package-lock.json to v2 hathor-wallet-service#75 - @andreabadesso
• [Partial] chore(deps): bump decode-uri-component from 0.2.0 to 0.2.2 hathor-wallet-service#80 - @luislhl
  • Status on Operational Excellence Epic: https://github.com/HathorNetwork/ops-tools/issues/36
• [Fail] feat: realiable integrations [part 1 - machine] hathor-wallet-service#89 - @andreabadesso
  • Started the week working on this but had to stop to work on the unplanned issues and the necessary changes for the token actions API
• [Fail] feat: the daemon should send the voided transactions when received from the fullnode hathor-wallet-service#8 - @andreabadesso
  • Same as above
• [Done] Investigate if we need to refactor the way we merge balances hathor-wallet-service#94 - @r4mmer
• [In Review] fix: handling throwing lambda call hathor-wallet-service#84 - @r4mmer

Unplanned issues:

• [Done] fix: real time txs + push notification hathor-wallet-service#104 - @andreabadesso
• [Done] fix: get token from decoded data hathor-wallet-service#14 - @andreabadesso
• [Done] feat: reliable integrations [part 2] hathor-wallet-service#90 - @andreabadesso
• [Done] bug: sync stopped in the new reliable integrations because of an ordering issue hathor-wallet-service#93 - @andreabadesso
• [Done] fix: ignoring NFT transaction outputs with undecoded scripts hathor-wallet-service#15 - @andreabadesso
• [Done] refactor: remove unused lambdas hathor-wallet-service#102 - @andreabadesso
• [Done] https://github.com/HathorNetwork/ops-tools/pull/57 - @luislhl
• [Done] https://github.com/HathorNetwork/ops-tools/pull/58 - @luislhl
• [Done] https://github.com/HathorNetwork/ops-tools/pull/60 - @luislhl
• [Done] https://github.com/HathorNetwork/ops-tools/pull/59 - @luislhl

Q3 Week 1: Jul 5th - Jul 9th

• [Partial] feat: add support for new token actions in the wallet lib hathor-wallet-lib#197 - @pedroferreira1
• [Fail] feat: wallet service integration hathor-wallet-mobile#101 - @pedroferreira1
• [Fail] Implement support for wallet service rollout hathor-wallet#208 - @pedroferreira1
• [Done] Debug wallet-service high costs with @luislhl - @andreabadesso
• [Partial] chore(deps): bump json5, @size-limit/preset-small-lib and tsconfig-paths hathor-wallet-service#82 - @andreabadesso
• [Partial] chore: updated package-lock.json to v2 hathor-wallet-service#75 - @andreabadesso
• [Partial] chore(deps): bump decode-uri-component from 0.2.0 to 0.2.2 hathor-wallet-service#80 - @andreabadesso
• [Fail] feat: realiable integrations [part 1 - machine] hathor-wallet-service#89 - @andreabadesso
• [Fail] feat: the daemon should send the voided transactions when received from the fullnode hathor-wallet-service#8 - @andreabadesso
• [Partial] Investigate if we need to refactor the way we merge balances hathor-wallet-service#94 - @r4mmer
• [Partial] fix: handling throwing lambda call hathor-wallet-service#84 - @r4mmer

Q2 Week 13: Jun 28th-Jul 02nd

• [Done] Deploy a new wallet-service and daemon instance to sync with the mainnet - @luislhl
• [Fail] Test the re-org strategy with more simulation strategies on the wallet service - @andreabadesso
• [Done] chore(deps): bump json5 from 1.0.1 to 1.0.2 hathor-wallet-service#83 - @andreabadesso
• [Partial] chore(deps): bump json5, @size-limit/preset-small-lib and tsconfig-paths hathor-wallet-service#82 - @andreabadesso
• [Partial] chore: updated package-lock.json to v2 hathor-wallet-service#75 - @andreabadesso
• [Done] chore: added codecov action and config hathor-wallet-service#74 - @andreabadesso
• [Partial] feat: the daemon should send the voided transactions when received from the fullnode hathor-wallet-service#8 - @andreabadesso
• [Partial] feat: add support for new token actions in the wallet lib hathor-wallet-lib#197 - @pedroferreira1
• [Fail] feat: wallet service integration hathor-wallet-mobile#101 - @pedroferreira1
• [Done] Mempool reviews for Full-Node api and Daemon - @r4mmer
• [Done] chore(deps): bump qs from 6.5.2 to 6.5.3 hathor-wallet-service#81 - @r4mmer
• [Done] chore(deps): bump minimist from 1.2.5 to 1.2.6 hathor-wallet-service#69 - @r4mmer

Q2 Week 12: Jun 21th-Jun 25th

• [Done] Send transactions before sending the block on daemon - @andreabadesso
• [Partial] Test the wallet service re-org strategy - @andreabadesso
• [Partial] New design/PoC Token Actions & implementation - @andreabadesso
• [Fail] Daemon: Sync voided transactions - @andreabadesso
• [Partial] design: OPS - Rate Limit, Alerts, Monitoring, QA, Deployment Steps - @andreabadesso
• [Fail] Add support for new token actions in the wallet lib - @pedroferreira1
• [Fail] Add support in wallet mobile for wallet service - @pedroferreira1
• [In Review] Mempool reviews for Full-Node api and Daemon - @r4mmer
• [In Review] Wallet Service: Finish out-of-order and reviews - @r4mmer
• [Blocked] Wallet Service: Test Authorization with new hathor-wallet-lib version - @r4mmer

Q2 Week 11: Jun 14th-Jun 18th

• [Partial] Test the wallet service re-org strategy - @andreabadesso
• [Partial] New design/PoC Token Actions & implementation - @andreabadesso
• [Fail] Daemon: Sync voided transactions - @andreabadesso
• [Partial] design: OPS - Rate Limit, Alerts, Monitoring, QA, Deployment Steps - @andreabadesso
• [Done] - Deploy the wallet-service and the daemon to the staging server with the latest handle-reorg merges - @andreabadesso
• [Done] - Improve the serverless deployment by creating the SQS queue automatically - @andreabadesso
• [Done] Wallet Service Daemon: sync mempool - @r4mmer
• [Done] Full-Node: mempool api - @r4mmer
• [Partial] Wallet Service: handle out-of-order mempool TXs - @r4mmer
• [Done] Finish wallet rollout design decision (https://github.com/HathorNetwork/internal-issues/issues/11) - @pedroferreira1
• [Fail] Add support for new token actions in the wallet lib - @pedroferreira1
• [Fail] Add support in wallet mobile for wallet service (feat: wallet service integration hathor-wallet-mobile#101) - @pedroferreira1

Q2 Week 10: Jun 7th-Jun 11th

• [Done] Wallet Service: Handle voided tx design and implementation - @andreabadesso
• [Fail] Wallet Service: Test the wallet service re-org strategy - @andreabadesso
• [Partial] Wallet Service: New design/PoC Token Actions & implementation - @andreabadesso
• [Fail] Wallet Service: handle out-of-order mempool TXs - @r4mmer
• [Partial] Full-Node: mempool api - @r4mmer
• [Partial] Daemon: sync mempool - @r4mmer

Q2 Week 9: May 31st-Jun 4th

• [Partial] Wallet Service: Handle voided tx design and implementation
• [Partial] Wallet Service: Test the wallet service re-org strategy
• [Fail] Wallet Service: New design/PoC Token Actions & implementation

Design Issues

• General Design (replaces feat: initial daemon implementation hathor-wallet-service#1)
• Mempool Sync
• Wallet Service Authentication
• Possible solutions for handling reorgs and balance recalculation
• Update wallets in real-time (guide-level explanation)
• Sync with the full-node

Pending issues to fulfill the design:

• Return the full derivation path for inputs on POST /txProposal
• TxProcessor should be idempotent
• POST /wallet needs a lock system to prevent errors with two requests arriving simultaneously
• TxHistory should receive AfterAddress and Count instead of Skip and Count
• feat: handle reorg

Pending tasks breakdown

• feat: send transactions in topological order
• feat: ignoring genesis transactions
• refactor: using the transactions table to handle blocks
• feat: handle timelock balance
• feat: tx-proposal code (to send txs)
• feat: handle authority outputs
• feat: tx-proposal token actions Design: fix: get token from decoded data hathor-wallet-service#14 PR: Release v1.4.0-alpha hathor-wallet-service#61
• refactor: database ORM (like sequelize) - [estimated work: 24h]
• refactor: database transaction/rollback support and error handling - This will come "free" from the ORM if we decide to use SequelizeJS - [estimated: 8h]
• design: Wallet Sync and Voided Txs Strategy - [estimated work: 40h]
  • Implementation: chore: add codecov into our CI pipeline hathor-wallet-service#72 (Currently in progress by @r4mmer)
• Develop a daemon that will listen for full node WebSocket messages and trigger block sync
  • Method that will sync blocks from a height range
  • Reorg strategy
  • Balance recalculation on reorgs - 5h chore: bump to v1.4.0 alpha hathor-wallet-service#60 release: v1.4.1-alpha hathor-wallet-service#64 (Currently in review)
    • Invoke the re-org from the daemon on the reorg state - 2h
    • Manually test the address_balance recalculation on multiple tokenIds - 2h
    • Increase coverage on the wallet-service - 1h
    • Automated tests using the hathor-core simulator - 8h
  • Improve daemon tests - 4h
• feat: sync mempool hathor-wallet-service#11
• feat: handle voided txs (Currently in progress by @andreabadesso)
  • New route to receive UPDATE_TX events - 2h
  • The daemon should send the UPDATE_TX events - 3h
• refactor: return correct status codes for errors and sucesses - [estimated work: 6h]
• docs: open API docs - [estimated work: 4h]
• feat: [Needs Design] wallet authentication - [estimated work includ. design: 40h] (Currently in progress by @r4mmer)
• chore: test with large wallets (20k+ txs) - [estimated work: 6h]
• refactor: optimize database (add indexes and use binary columns) - [estimated work: 16h]
• design: OPS - Rate Limit, Alerts, Monitoring, QA, Deployment Steps - [estimated work: 16h]
• design: use the wallet service in the wallets - [estimated work: 40h for the wallets desktop and mobile]
  • Finish wallet rollout low level design
  • Add support for new token actions in the wallet lib
  • Add support in wallet mobile for wallet service
  • Add support for all wallets with rollout logic

Summary

Total: 336h / 42 days / 8.4 weeks, i.e., end of Q1/2021.

[andreabadesso]

Here is the estimated work for the breakdown tasks:

handle timelock balance
tx-proposal code (to send txs)
handle authority outputs
tx-proposal code (token actions) - [estimated work: 40h]
handle voided txs - [estimated work: 12h]
database transaction/rollback support and error handling - This will come "free" from the ORM if we decide to use SequelizeJS
database ORM (like sequelize) - [estimated work: 24h]
optimize database (add indexes and use binary columns) - [estimated work: 16h]
open API docs - [estimated work: 8h]
wallet sync strategy (websocket?) - [estimated work: 40h]
[Needs Design] wallet authentication - [estimated work includ. design: 40h]
test with large wallets (20k+ txs) - [estimated work: 6h]

[andreabadesso]

Q3 Planning

Q2 spill-overs

• feat: TxHistory should receive AfterAddress and Count instead of Skip and Count - 8h
• feat: POST /wallet needs a lock system to prevent two requests arriving simultaneously - 12h
• refactor: Database ORM (like sequelize) - 24h
• refactor: Database transaction/rollback support and error handling - 8h
• chore: Manually test the address_balance recalculation on multiple tokenIds - 4h
• chore: Automated tests using the hathor-core simulator - 8h
• feat: Handle voided transactions on wallet-service daemon - 8h
• feat: Handle voided transactions on wallet-service - 8h
• refactor: Optimize database (add indexes and use binary columns) - 12h
• design: OPS - Rate Limit, Alerts, Monitoring, QA, Deployment Steps - 8h
• feat: Authorization for the websocket connection - 12h

Total: 112h (~3 weeks, includ. current)

Q3

• chore: database healthcheck to be able to check if state is consistent with the fullnode - 24h
• feat: Multisig support on txProposalCreate - 32h
• feat: Nano Contracts support - ?h
• feat: Atomic Swap support - ?h

Q3 - Data Service