Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The output_pos index not being cleaned up correctly during compaction #2606

Closed
antiochp opened this issue Feb 19, 2019 · 1 comment
Closed
Assignees
Labels
Milestone

Comments

@antiochp
Copy link
Member

See #2603

We read raw bytes from the data file during the pruning/compaction process and use these bytes to clean the output_pos index up.

  1. For each pruned chunk of bytes we attempt to call delete_output_pos but these bytes do not simply represent a commitment, they represent an output_identifier.

  2. The chunk of bytes is actually empty...

  3. It is not actually safe to do this as we can have duplicate outputs in the TXO set if one is spent and one is unspent. And in this scenario it is not safe to clean the output_pos index up based purely on spent outputs.

@antiochp antiochp added the bug label Feb 19, 2019
@antiochp antiochp added this to the 1.0.2 milestone Feb 19, 2019
@antiochp antiochp self-assigned this Feb 19, 2019
@antiochp
Copy link
Member Author

antiochp commented Feb 19, 2019

I'm kind of tempted to get rid of the callback mechanism entirely and just brute force it via rebuild_index every time we compact the chain.

See #2607

Or do something like delete_peers() in p2p store where we iterate over entries in the db based on prefix and remove those that meet some defined criteria (in this case spent outputs).

This isn't really any more "brute force" than how we currently iterate over the entire data file pruning as we go.
We'd basically just do two passes, one to prune the file and then another pass over the output_pos index (which grows with the UTXO set) to check if the entries are still valid and removing them where necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants