Storing intermediate state (archive mode)

[lrettig]

As discussed in today's design review, and related to spacemeshos/SMIPS#13, we need to decide which global state data are stored in a go-spacemesh node by default, and how "archive mode" data such as intermediate state are stored (e.g., in process in go-spacemesh or in a separate "archive tool" process), and when and how these data are pruned by a node.

@noamnelke argued that a full node should store only the data required by a normal user for normal daily usage, e.g., all transactions (and receipts) that touch their account(s). He suggested that the node API make available streams of all relevant data, and otherwise be kept as simple as possible, and that a separate process can consume and store these data.

@avive argued that, in practice, users are likely to create additional accounts in the future, so this sort of solution is impractical as it would require replaying everything from genesis to, e.g., get the balance of a new account.

[avive]

We talked about it several times - we must support wallets which are restored and want to see recent transactions and we don't know in advance in the node which accounts these should be. I thought we agreed that the archive node stores everything and full node only store non accounts/apps db global state data for n layer in the past so it is a good tradeoff between connivence and data storage requirements. Are we good good on this or do we need to further discuss?

[lrettig]

@noamnelke was making the case that there should be no "archive mode" for go-spacemesh - that it should be a separate, overlay process of some sort. So I think we need to reach consensus on this point at least, because it has major implications for architecture.

[lrettig]

For the record, I don't think the two ideas are incompatible: you could have a full node store only the data that the user would need to run a wallet app, and have an overlay, module, or something else that stores other data.

[avive]

I think there are 2 aspects to archive vs. full: 1. store old global state data which is not part of accounts db that full nodes might not keep (receipts, events, rewards, etc...). 2. store indexes and data that enable to query the global state at any layer - something that a full node doesn't provide.

[lrettig]

I agree on the second. The first is a little more contentious: you could make the case (and I think you'd agree) that this is data that any full node might need to service any account - for the reason you state - that if someone adds/restores an account later the wallet would still need to be able to display old rewards, etc

[avive]

Yes, but what I'm saying is that there will be a hard-coded per network param which indicates the n=1000 layers from the tip that a full node keeps this non-accounts-db state for use of wallets. This is a reasnoable tradeoff. If you resotre a wallet on your local node you will get all the relevent data going back some time but really old rewards and transactions will not be available and users should use an explorer to view these.

I also think that this decision should be completely transperant to api clients and that no api changes are needed. If you ask for data that the node doesn't have (because it pruned it as too old) it will just not return results but if the api provided is an archive node then you'll get the data back. So this issue is more about how to implement storage of some global state data (non-accounts-db) in nodes based on whether they are configured as archive or just full nodes....

[lrettig]

there will be a hard-coded per network param which indicates the n=1000 layers

Why make this a "network param"? It has nothing to do with the network or with consensus, right? It seems like something that the user would want to configure for themselves.

If you resotre a wallet on your local node you will get all the relevent data going back some time but really old rewards and transactions will not be available and users should use an explorer to view these.

I see what you mean. Don't you think this might make users think that their balance, and tx history, hasn't been fully validated by their node, though? (Even if this is not the case.)

I also think that this decision should be completely transperant to api clients and that no api changes are needed

I think the API needs a way to indicate that a request looks valid but that it doesn't currently know the answer, or have the data. See spacemeshos/api#50.

[lrettig]

Linking spacemeshos/go-spacemesh#1985 here