Extract stake pool registration from on-chain data #1761
Assignees
Milestone
Comments
iohk-bors bot
added a commit
that referenced
this issue
Jun 18, 2020
1763: extract stake pool registrations r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1761 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - a57cf75 📍 **extend pool database with two new tables to store metadata** one of them will be used to persist a queue of metadata to fetch, while the other is a less volatile storage where metadata gets dumped once fetched. There's in principle no need for caching or timestamps here since metadata can't be updated (hash is stored on-chain as part of the registration) which simplifies a bit the problem. - e776d1c 📍 **extract pool registration certificate from blocks** - 29166e4 📍 **add logic for monitoring shelley stake pools registration certificates** Still rudimentary, and doesn't keep track of metadata information _yet_ but it's coming. - 1331015 📍 **write stake pool monitoring in start-up code, with proper loggers** - 7e62661 📍 **destroy node's connection when they're no longer needed** - 6065134 📍 **do not force rollback when the list of checkpoints is empty** An empty list basically assumes that there's no checkpoint on the driver, so nothing to rollback. - f49a7bf 📍 **extract and store pool metadata references from the chain** - 4ef1b80 📍 **add basic property tests for checking new pool db functions.** - 6ef5980 📍 **store block production in the pool monitoring worker** - e8e1d83 📍 **change database fn for fetching metadata ref to return them one-by-one** This is how the worker uses it in the end, and it makes the signature looks less awkward. - 2c63840 📍 **avoid intermediary akward 'metadataQueue' table** Instead, we can store metadata references alongside the registration (with blank / default values for Jörmungandr) - 279cd62 📍 **remove unused module 'Cardano.Pool.Metadata'** Turns out that we won't be able to fetch metadata from Smash as it was originally intended: a) Metadata are fetchted by metadata hash b) Metadata needs to be fetched as raw bytestring, and decoded to JSON _after_ so that we are able to compute their corresponding hash and verify it. If we immediately deserialize to JSON, then we would have to re-encode them but there's no particular canonical encoding enforced when registering metadata, which leaves many possibility about how to encode a single file. Plus, there's no need for caching at the request-level since once fetched, there's no reason to fetch a result again. This module is therefore useless. - d393b7d 📍 **move jormungandr-specific data-type into jormungandr package target** This cleans up a bit the metadata-related types because it had become a mess. In the end, we'll keep in the core package the types as they're expected by cardano-wallet-shelley and simply move the exceptional ones (jormungandr) inside its relevant package. # Comments <!-- Additional comments or screenshots to attach if any --> Kinda hard to fully test without wiring it up into the stake pool listing, but logs are quite positive: ``` [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.17 UTC] Monitoring stake pools. Currently at genesis [cardano-wallet.network:Info:1071] [2020-06-17 10:22:36.17 UTC] Looking for an intersection with the node's local chain with: [cardano-wallet.network:Info:1071] [2020-06-17 10:22:36.18 UTC] Intersection found: 01010101 [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.18 UTC] Applying blocks [0.0 ... 0.0] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Applying blocks [0.12 ... 2.0] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of 5a7b67c7 owned by [ed25519_pk1h3cqm8h6glw56hqa5mgy0gl97jxcawzpphxy5hwtchq8hra6zcwq70fndy] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of c7258ccc owned by [ed25519_pk1mnjxa09v9hmn48awwcgyrfmnnq4dvs7fsege3glcg6yn0vckn0xsrm4zqy] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of 775af3b2 owned by [ed25519_pk1qud4ay0zapd0r43c88lc9av8fnesx2hu0r36dsf46lprdp9p3zksx7q880] ``` <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]> Co-authored-by: IOHK <[email protected]> Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot
added a commit
that referenced
this issue
Jun 18, 2020
1763: extract stake pool registrations r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1761 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - a57cf75 📍 **extend pool database with two new tables to store metadata** one of them will be used to persist a queue of metadata to fetch, while the other is a less volatile storage where metadata gets dumped once fetched. There's in principle no need for caching or timestamps here since metadata can't be updated (hash is stored on-chain as part of the registration) which simplifies a bit the problem. - e776d1c 📍 **extract pool registration certificate from blocks** - 29166e4 📍 **add logic for monitoring shelley stake pools registration certificates** Still rudimentary, and doesn't keep track of metadata information _yet_ but it's coming. - 1331015 📍 **write stake pool monitoring in start-up code, with proper loggers** - 7e62661 📍 **destroy node's connection when they're no longer needed** - 6065134 📍 **do not force rollback when the list of checkpoints is empty** An empty list basically assumes that there's no checkpoint on the driver, so nothing to rollback. - f49a7bf 📍 **extract and store pool metadata references from the chain** - 4ef1b80 📍 **add basic property tests for checking new pool db functions.** - 6ef5980 📍 **store block production in the pool monitoring worker** - e8e1d83 📍 **change database fn for fetching metadata ref to return them one-by-one** This is how the worker uses it in the end, and it makes the signature looks less awkward. - 2c63840 📍 **avoid intermediary akward 'metadataQueue' table** Instead, we can store metadata references alongside the registration (with blank / default values for Jörmungandr) - 279cd62 📍 **remove unused module 'Cardano.Pool.Metadata'** Turns out that we won't be able to fetch metadata from Smash as it was originally intended: a) Metadata are fetchted by metadata hash b) Metadata needs to be fetched as raw bytestring, and decoded to JSON _after_ so that we are able to compute their corresponding hash and verify it. If we immediately deserialize to JSON, then we would have to re-encode them but there's no particular canonical encoding enforced when registering metadata, which leaves many possibility about how to encode a single file. Plus, there's no need for caching at the request-level since once fetched, there's no reason to fetch a result again. This module is therefore useless. - d393b7d 📍 **move jormungandr-specific data-type into jormungandr package target** This cleans up a bit the metadata-related types because it had become a mess. In the end, we'll keep in the core package the types as they're expected by cardano-wallet-shelley and simply move the exceptional ones (jormungandr) inside its relevant package. # Comments <!-- Additional comments or screenshots to attach if any --> Kinda hard to fully test without wiring it up into the stake pool listing, but logs are quite positive: ``` [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.17 UTC] Monitoring stake pools. Currently at genesis [cardano-wallet.network:Info:1071] [2020-06-17 10:22:36.17 UTC] Looking for an intersection with the node's local chain with: [cardano-wallet.network:Info:1071] [2020-06-17 10:22:36.18 UTC] Intersection found: 01010101 [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.18 UTC] Applying blocks [0.0 ... 0.0] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Applying blocks [0.12 ... 2.0] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of 5a7b67c7 owned by [ed25519_pk1h3cqm8h6glw56hqa5mgy0gl97jxcawzpphxy5hwtchq8hra6zcwq70fndy] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of c7258ccc owned by [ed25519_pk1mnjxa09v9hmn48awwcgyrfmnnq4dvs7fsege3glcg6yn0vckn0xsrm4zqy] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of 775af3b2 owned by [ed25519_pk1qud4ay0zapd0r43c88lc9av8fnesx2hu0r36dsf46lprdp9p3zksx7q880] ``` <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]> Co-authored-by: IOHK <[email protected]> Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot
added a commit
that referenced
this issue
Jun 18, 2020
1763: extract stake pool registrations r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1761 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - a57cf75 📍 **extend pool database with two new tables to store metadata** one of them will be used to persist a queue of metadata to fetch, while the other is a less volatile storage where metadata gets dumped once fetched. There's in principle no need for caching or timestamps here since metadata can't be updated (hash is stored on-chain as part of the registration) which simplifies a bit the problem. - e776d1c 📍 **extract pool registration certificate from blocks** - 29166e4 📍 **add logic for monitoring shelley stake pools registration certificates** Still rudimentary, and doesn't keep track of metadata information _yet_ but it's coming. - 1331015 📍 **write stake pool monitoring in start-up code, with proper loggers** - 7e62661 📍 **destroy node's connection when they're no longer needed** - 6065134 📍 **do not force rollback when the list of checkpoints is empty** An empty list basically assumes that there's no checkpoint on the driver, so nothing to rollback. - f49a7bf 📍 **extract and store pool metadata references from the chain** - 4ef1b80 📍 **add basic property tests for checking new pool db functions.** - 6ef5980 📍 **store block production in the pool monitoring worker** - e8e1d83 📍 **change database fn for fetching metadata ref to return them one-by-one** This is how the worker uses it in the end, and it makes the signature looks less awkward. - 2c63840 📍 **avoid intermediary akward 'metadataQueue' table** Instead, we can store metadata references alongside the registration (with blank / default values for Jörmungandr) - 279cd62 📍 **remove unused module 'Cardano.Pool.Metadata'** Turns out that we won't be able to fetch metadata from Smash as it was originally intended: a) Metadata are fetchted by metadata hash b) Metadata needs to be fetched as raw bytestring, and decoded to JSON _after_ so that we are able to compute their corresponding hash and verify it. If we immediately deserialize to JSON, then we would have to re-encode them but there's no particular canonical encoding enforced when registering metadata, which leaves many possibility about how to encode a single file. Plus, there's no need for caching at the request-level since once fetched, there's no reason to fetch a result again. This module is therefore useless. - d393b7d 📍 **move jormungandr-specific data-type into jormungandr package target** This cleans up a bit the metadata-related types because it had become a mess. In the end, we'll keep in the core package the types as they're expected by cardano-wallet-shelley and simply move the exceptional ones (jormungandr) inside its relevant package. # Comments <!-- Additional comments or screenshots to attach if any --> Kinda hard to fully test without wiring it up into the stake pool listing, but logs are quite positive: ``` [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.17 UTC] Monitoring stake pools. Currently at genesis [cardano-wallet.network:Info:1071] [2020-06-17 10:22:36.17 UTC] Looking for an intersection with the node's local chain with: [cardano-wallet.network:Info:1071] [2020-06-17 10:22:36.18 UTC] Intersection found: 01010101 [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.18 UTC] Applying blocks [0.0 ... 0.0] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Applying blocks [0.12 ... 2.0] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of 5a7b67c7 owned by [ed25519_pk1h3cqm8h6glw56hqa5mgy0gl97jxcawzpphxy5hwtchq8hra6zcwq70fndy] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of c7258ccc owned by [ed25519_pk1mnjxa09v9hmn48awwcgyrfmnnq4dvs7fsege3glcg6yn0vckn0xsrm4zqy] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of 775af3b2 owned by [ed25519_pk1qud4ay0zapd0r43c88lc9av8fnesx2hu0r36dsf46lprdp9p3zksx7q880] ``` <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]> Co-authored-by: IOHK <[email protected]> Co-authored-by: Johannes Lund <[email protected]>
iohk-bors bot
added a commit
that referenced
this issue
Jun 18, 2020
1763: extract stake pool registrations r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1761 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - a57cf75 📍 **extend pool database with two new tables to store metadata** one of them will be used to persist a queue of metadata to fetch, while the other is a less volatile storage where metadata gets dumped once fetched. There's in principle no need for caching or timestamps here since metadata can't be updated (hash is stored on-chain as part of the registration) which simplifies a bit the problem. - e776d1c 📍 **extract pool registration certificate from blocks** - 29166e4 📍 **add logic for monitoring shelley stake pools registration certificates** Still rudimentary, and doesn't keep track of metadata information _yet_ but it's coming. - 1331015 📍 **write stake pool monitoring in start-up code, with proper loggers** - 7e62661 📍 **destroy node's connection when they're no longer needed** - 6065134 📍 **do not force rollback when the list of checkpoints is empty** An empty list basically assumes that there's no checkpoint on the driver, so nothing to rollback. - f49a7bf 📍 **extract and store pool metadata references from the chain** - 4ef1b80 📍 **add basic property tests for checking new pool db functions.** - 6ef5980 📍 **store block production in the pool monitoring worker** - e8e1d83 📍 **change database fn for fetching metadata ref to return them one-by-one** This is how the worker uses it in the end, and it makes the signature looks less awkward. - 2c63840 📍 **avoid intermediary akward 'metadataQueue' table** Instead, we can store metadata references alongside the registration (with blank / default values for Jörmungandr) - 279cd62 📍 **remove unused module 'Cardano.Pool.Metadata'** Turns out that we won't be able to fetch metadata from Smash as it was originally intended: a) Metadata are fetchted by metadata hash b) Metadata needs to be fetched as raw bytestring, and decoded to JSON _after_ so that we are able to compute their corresponding hash and verify it. If we immediately deserialize to JSON, then we would have to re-encode them but there's no particular canonical encoding enforced when registering metadata, which leaves many possibility about how to encode a single file. Plus, there's no need for caching at the request-level since once fetched, there's no reason to fetch a result again. This module is therefore useless. - d393b7d 📍 **move jormungandr-specific data-type into jormungandr package target** This cleans up a bit the metadata-related types because it had become a mess. In the end, we'll keep in the core package the types as they're expected by cardano-wallet-shelley and simply move the exceptional ones (jormungandr) inside its relevant package. # Comments <!-- Additional comments or screenshots to attach if any --> Kinda hard to fully test without wiring it up into the stake pool listing, but logs are quite positive: ``` [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.17 UTC] Monitoring stake pools. Currently at genesis [cardano-wallet.network:Info:1071] [2020-06-17 10:22:36.17 UTC] Looking for an intersection with the node's local chain with: [cardano-wallet.network:Info:1071] [2020-06-17 10:22:36.18 UTC] Intersection found: 01010101 [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.18 UTC] Applying blocks [0.0 ... 0.0] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Applying blocks [0.12 ... 2.0] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of 5a7b67c7 owned by [ed25519_pk1h3cqm8h6glw56hqa5mgy0gl97jxcawzpphxy5hwtchq8hra6zcwq70fndy] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of c7258ccc owned by [ed25519_pk1mnjxa09v9hmn48awwcgyrfmnnq4dvs7fsege3glcg6yn0vckn0xsrm4zqy] [cardano-wallet.pools-engine:Info:1071] [2020-06-17 10:22:36.19 UTC] Discovered stake pool registration: Registration of 775af3b2 owned by [ed25519_pk1qud4ay0zapd0r43c88lc9av8fnesx2hu0r36dsf46lprdp9p3zksx7q880] ``` <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]> Co-authored-by: IOHK <[email protected]> Co-authored-by: Johannes Lund <[email protected]>
|
lgtm |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Context
As a wallet user I want to Check stakepool metadata so that I can make an informed delegation choice.
Stake pool metadata are registered on chain inside stake pool registrations alongside some interesting data like the margin, cost or pledge amount. These registrations can be parsed and the information inside them extracted as we crawl the chain so that we can access it later.
Decision
Acceptance Criteria
Development
QA
This is relatively hard to test in isolation because to get proper testing, we need to somehow simulate a blockchain with registration and simulate interesting cases. Ultimately, we'll be able to confirm that this works as intended with e2e tests in the integration scenarios, although we need to finalize stake pools listing first (see Use and combine chain data with LSQ data for stake pools #1769)
The PR piggy-back on existing tables and interfaces developed for the ITN, and tests written at this time still apply:
The text was updated successfully, but these errors were encountered: