-
Notifications
You must be signed in to change notification settings - Fork 697
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to query archival data with recent upgrade 14470501 #2313
Comments
Thank you for opening the issue! Would you mind sharing your node's configuration and please confirm that pruning is turned off in # default: the last 362880 states are kept, pruning at 10 block intervals
# nothing: all historic states will be saved, nothing will be deleted (i.e. archiving node)
# everything: 2 latest states will be kept; pruning at 10 block intervals.
# custom: allow pruning options to be manually specified through 'pruning-keep-recent', and 'pruning-interval'
pruning = "nothing" This configuration should allow your node to keep all historic blocks. |
Yup, pruning is disabled:
|
Here is the same issue.
Config set up is pruning nothing. So it is archive node. |
I've found a hacky fix by patching the cosmos-sdk as follows:
I'm not sure of all implications but I guess this is a regression that was introduced in the SDK bump during the Gaia v8 upgrade. |
This affected Juno as well, quoting Reece:
So adding the new storeKey is the way forward, this is backwards compatible. I'll try to come up with a cleaner patch when I find some time if that sounds good (unless it's solved before :)). |
I'd surely say it's a regression. |
Validator feedback: If this is how the system works, then for historical reporting purposes looks like we are being asked to run a dedicated node with older binaries until upgrade heights for every single cosmos release ? |
I can confirm this resolved the issue temporarily for me. Modify
then
|
@MSalopek Still this issue on going? |
Hey there, We have written a patch. You can find the patched version in the pull requests. We have submitted the patch to the cosmos SDK. Cosmo's SDK 45 is deprecated. The Cosmos SDK team simultaneously will not merge the patch to 45 and recommends against using a fork. |
## Description Closes: [cosmos/gaia#2313](cosmos/gaia#2313) The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade. I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... * [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] added `!` to the type prefix if API or client breaking change * [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting)) * [ ] provided a link to the relevant issue or specification * [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules) * [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing) * [ ] added a changelog entry to `CHANGELOG.md` * [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) * [ ] updated the relevant documentation or specification * [ ] reviewed "Files changed" and left comments if necessary * [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... * [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] confirmed `!` in the type prefix if API or client breaking change * [ ] confirmed all author checklist items have been addressed * [ ] reviewed state machine logic * [ ] reviewed API design and naming * [ ] reviewed documentation is accurate * [ ] reviewed tests and test coverage * [ ] manually tested (if applicable)
## Description Closes: [cosmos/gaia#2313](cosmos/gaia#2313) The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade. I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... * [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] added `!` to the type prefix if API or client breaking change * [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting)) * [ ] provided a link to the relevant issue or specification * [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules) * [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing) * [ ] added a changelog entry to `CHANGELOG.md` * [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) * [ ] updated the relevant documentation or specification * [ ] reviewed "Files changed" and left comments if necessary * [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... * [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] confirmed `!` in the type prefix if API or client breaking change * [ ] confirmed all author checklist items have been addressed * [ ] reviewed state machine logic * [ ] reviewed API design and naming * [ ] reviewed documentation is accurate * [ ] reviewed tests and test coverage * [ ] manually tested (if applicable) (cherry picked from commit d5a618d) # Conflicts: # store/CHANGELOG.md
## Description Closes: [cosmos/gaia#2313](cosmos/gaia#2313) The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade. I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... * [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] added `!` to the type prefix if API or client breaking change * [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting)) * [ ] provided a link to the relevant issue or specification * [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules) * [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing) * [ ] added a changelog entry to `CHANGELOG.md` * [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) * [ ] updated the relevant documentation or specification * [ ] reviewed "Files changed" and left comments if necessary * [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... * [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] confirmed `!` in the type prefix if API or client breaking change * [ ] confirmed all author checklist items have been addressed * [ ] reviewed state machine logic * [ ] reviewed API design and naming * [ ] reviewed documentation is accurate * [ ] reviewed tests and test coverage * [ ] manually tested (if applicable) (cherry picked from commit d5a618d) # Conflicts: # store/CHANGELOG.md
@mmulji-ic @MSalopek @faddat Any update please? |
Hi @yj0x0x checking, unfortunately we don't have the infra setup, so it takes some time. Dan mentioned a patch above, for the SDK that you can also try. We've talked to the SDK folks and they won't backport that to v0.45 line unfortunately. |
Closes: [cosmos/gaia#2313](cosmos/gaia#2313) The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade. I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height. --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... * [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] added `!` to the type prefix if API or client breaking change * [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting)) * [ ] provided a link to the relevant issue or specification * [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules) * [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing) * [ ] added a changelog entry to `CHANGELOG.md` * [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) * [ ] updated the relevant documentation or specification * [ ] reviewed "Files changed" and left comments if necessary * [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... * [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] confirmed `!` in the type prefix if API or client breaking change * [ ] confirmed all author checklist items have been addressed * [ ] reviewed state machine logic * [ ] reviewed API design and naming * [ ] reviewed documentation is accurate * [ ] reviewed tests and test coverage * [ ] manually tested (if applicable)
* feat: More flexibility for `CacheMultiStoreWithVersion` (cosmos#15683) Closes: [cosmos/gaia#2313](cosmos/gaia#2313) The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade. I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height. --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... * [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] added `!` to the type prefix if API or client breaking change * [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting)) * [ ] provided a link to the relevant issue or specification * [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules) * [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing) * [ ] added a changelog entry to `CHANGELOG.md` * [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) * [ ] updated the relevant documentation or specification * [ ] reviewed "Files changed" and left comments if necessary * [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... * [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] confirmed `!` in the type prefix if API or client breaking change * [ ] confirmed all author checklist items have been addressed * [ ] reviewed state machine logic * [ ] reviewed API design and naming * [ ] reviewed documentation is accurate * [ ] reviewed tests and test coverage * [ ] manually tested (if applicable) * fix merge conflict
We're planing to upgrade Gaia to SDK 0.47 in the following months. In the meantime, please use the suggested fix. |
|
The suggested fix doesn't work anymore. |
Hi @Jacoblava @gilxgil we'll take a look to see if we can adapt that fix or suggestion another mitigation. |
@mmulji-ic Is there any update / fix / patch for it? |
@Jacoblava I have updated my unofficial patch https://github.com/Reecepbcups/cosmos-sdk/releases/tag/v0.45.16-ics-lsm-iavlpatch.1 replace github.com/cosmos/cosmos-sdk => github.com/reecepbcups/cosmos-sdk v0.45.16-ics-lsm-iavlpatch.1 also cc @mmulji-ic |
oh and also @gilxgil ^ |
Thanks a lot,
Will look at it now.
…On Mon, Oct 2, 2023, 21:52 Reece Williams ***@***.***> wrote:
oh and also @gilxgil <https://github.com/gilxgil> ^
—
Reply to this email directly, view it on GitHub
<#2313 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/A4PBZQFH2YEEYNBUPEVCF73X5MEPFAVCNFSM6AAAAAAV65W7G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBTGU4DEMBZGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
@ScepticMatt Yea this lsm patch is only for Gaia. Axelar probably has their own fork then or they use Ethermint? My patches do not support this |
Thanks I saw that they forked it. I forked their fork and will try if I can bypass this |
@Jacoblava @ScepticMatt please repoen if your forked forks do not work. |
Closes: [cosmos/gaia#2313](cosmos/gaia#2313) The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade. I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height. --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... * [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] added `!` to the type prefix if API or client breaking change * [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting)) * [ ] provided a link to the relevant issue or specification * [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules) * [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing) * [ ] added a changelog entry to `CHANGELOG.md` * [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) * [ ] updated the relevant documentation or specification * [ ] reviewed "Files changed" and left comments if necessary * [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... * [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] confirmed `!` in the type prefix if API or client breaking change * [ ] confirmed all author checklist items have been addressed * [ ] reviewed state machine logic * [ ] reviewed API design and naming * [ ] reviewed documentation is accurate * [ ] reviewed tests and test coverage * [ ] manually tested (if applicable)
Closes: [cosmos/gaia#2313](cosmos/gaia#2313) The gaia v9 upgrade causes node to errors out when querying historical states before the upgrade. This is because we added a new module store in the upgrade which is the `provider module store`. Thus, every times we load the multi store for querying at a height prior to the upgrade, we try to load the `provider module store` and fail since it has not existed before the upgrade. I make changes to `CacheMultiStoreWithVersion` so that if we fail to load any module store at a given height, we check if that module store has existed at that height or not (using `getCommitInfo()`), we can ignore the error if the store hasn't existed at that given height. --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... * [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] added `!` to the type prefix if API or client breaking change * [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#pr-targeting)) * [ ] provided a link to the relevant issue or specification * [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/main/docs/docs/building-modules) * [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/main/CONTRIBUTING.md#testing) * [ ] added a changelog entry to `CHANGELOG.md` * [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) * [ ] updated the relevant documentation or specification * [ ] reviewed "Files changed" and left comments if necessary * [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... * [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] confirmed `!` in the type prefix if API or client breaking change * [ ] confirmed all author checklist items have been addressed * [ ] reviewed state machine logic * [ ] reviewed API design and naming * [ ] reviewed documentation is accurate * [ ] reviewed tests and test coverage * [ ] manually tested (if applicable)
Summary of Bug
Since the Gaia migration at block 14470501, historical data is screwed up for blocks below.
Version
gaiad-v9.0.1-linux-amd64
Steps to Reproduce
Blocks >= 14470501: OK
Blocks < 14470501: KO
This happens for all gRPC calls, no matter which module is used, using a fresh Quicksync full archival dump using
gaiad-v9.0.1-linux-amd64
version.Is this a regression?
For Admin Use
The text was updated successfully, but these errors were encountered: