Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add option to retain sat index for spent outputs #2999

Merged
merged 2 commits into from
Jan 15, 2024
Merged

Add option to retain sat index for spent outputs #2999

merged 2 commits into from
Jan 15, 2024

Conversation

casey
Copy link
Collaborator

@casey casey commented Jan 9, 2024

Add --index-spent-sats option to keep sat index entries for spent outputs.

This is how the index originally worked, but it was too expensive, taking forever to build and swelling the size of the index to over 10 terabytes, if I recall correctly.

This is untested, and is added for anyone who wants to try it out. It will likely require a crazily beefy machine to run, like 256GB+ of RAM, and 10s or 100s of terabytes of disk.

@casey casey marked this pull request as ready for review January 13, 2024 01:50
@casey
Copy link
Collaborator Author

casey commented Jan 13, 2024

I thought this would be easy, but it turned into a pretty big PR. The reason is that we had the weird List enum which both contained sat ranges, and indicated whether an output was spent or unspent. --index-spent-sats required revisiting that, since you could now have a spent output, but still have the sat ranges.

@casey casey merged commit 41d9474 into ordinals:master Jan 15, 2024
6 checks passed
@casey casey deleted the index-historical-sats branch January 15, 2024 20:16
lifofifoX added a commit to lifofifoX/ord that referenced this pull request Jan 16, 2024
@lifofifoX
Merge branch 'master' into inscription-recursive-endpoint
00651ae 
* master: (175 commits)
  Inscribe with delegate (ordinals#3021)
  Add blocks and transaction JSON endpoints (ordinals#3004)
  Use --name instead of --wallet in README (ordinals#3010)
  Hide BVM Network inscriptions (ordinals#3012)
  Add option to retain sat index for spent outputs (ordinals#2999)
  Don't use browser sniffing when serving favicon (ordinals#3003)
  Add minimal Dockerfile (ordinals#2786)
  Cache less aggressively (ordinals#3002)
  Add `indexed` to output JSON (ordinals#2971)
  Remove dead link from README (ordinals#3000)
  Add crate to audit content security policy (ordinals#2993)
  Optimize get_inscription_ids_by_sat_paginated (ordinals#2996)
  Suppress empty command output (ordinals#2995)
  Add recipe to deploy to all servers in fleet (ordinals#2992)
  Release 0.15.0 (ordinals#2989)
  Fix Project Board link (ordinals#2991)
  Make `FundRawTransactionOptions ` public (ordinals#2938)
  Use enums for runestone tags and flags (ordinals#2956)
  Update server names in justfile (ordinals#2954)
  Update delegate.md (ordinals#2976)
  ...
thedoublejay pushed a commit to sadoprotocol/ord that referenced this pull request Mar 12, 2024
@joundy @casey
@oxSaturn @raphjaph @RobertClarke @DaviRain-Su @JeremyRubin @mj10021 @zhiqiangxu @lugondev @jerryfane @HarveyV @mi-yu @lifofifoX @elocremarc @aekasitt @bingryan @andrewhong5297 @arik-so
bump(ord): version 0.16.0 (#5)
1b4ab29 
* Add recipe to deploy to all servers in fleet (ordinals#2992)

* Suppress empty command output (ordinals#2995)

* Optimize get_inscription_ids_by_sat_paginated (ordinals#2996)

* Add crate to audit content security policy (ordinals#2993)

* Remove dead link from README (ordinals#3000)

* Add `indexed` to output JSON (ordinals#2971)

* Cache less aggressively (ordinals#3002)

* Add minimal Dockerfile (ordinals#2786)

* Don't use browser sniffing when serving favicon (ordinals#3003)

* Add option to retain sat index for spent outputs (ordinals#2999)

* Hide BVM Network inscriptions (ordinals#3012)

* Use --name instead of --wallet in README (ordinals#3010)

* Add blocks and transaction JSON endpoints (ordinals#3004)

* Inscribe with delegate (ordinals#3021)

* Use untyped table API to get table info (ordinals#2747)

* Make wallet communicate with index via RPC (ordinals#2929)

* Break deploy recipes into multiple lines (ordinals#3026)

* Test fee-spent inscription numbering (ordinals#3032)

* Remove uneccessary allocations in Inscription Script Creation (ordinals#3039)

* Better wallet error messages (ordinals#3041)

* Add documentation for reinscriptions (ordinals#2963)

* Exclude unnecessary docs (ordinals#3043)

* Enable JSON API by default (ordinals#3047)

* Forbid destinations in same-sat mode (ordinals#3038)

* Dump and restore wallet from descriptors (ordinals#3048)

* Add /runes/balances (ordinals#2978)

* Make invariant message more concise (ordinals#3029)

* Add dry run to send, print Outgoing and PSBT (ordinals#3063)

* Use a flag to indicate a mint (ordinals#3068)

* Move SatPoint into library (ordinals#3077)

* [ordinals] Bump version: 0.0.1 → 0.0.2 (ordinals#3078)

* Use min instead of clamp (ordinals#3081)

* Make clippy stop complaining about insane repair callback (ordinals#3104)

* Remove index parameter from index_block (ordinals#3088)

* Allow specifying satpoint in `same-sat` batch inscribe (ordinals#3100)

* Show reinscriptions in `ord wallet inscriptions` (ordinals#3101)

* Move sat and friends into ordinals crate (ordinals#3079)

* fix naming (ordinals#3112)

* Return signed PSBT from `ord wallet send` (ordinals#3093)

* Add /r/blockinfo endpoint (ordinals#3075)

* Import multiple descriptors at a time (ordinals#3091)

* Add `satpoints` batch inscribe mode (ordinals#3115)

* Allow inscribing AVIF images (ordinals#3123)

* Fix spelling mistake in bip.mediawiki (ordinals#3118)

* Only allow mnemonic from stdin (ordinals#3023)

* Emit inscription update events to channel (ordinals#3137)

* Make Options public (ordinals#3138)

* Use `image-rendering: auto` when downscaling images (ordinals#3144)

* Add `ord env` to spin up a test bitcoin daemon and ord server (ordinals#3146)

* Display inscription content type counts on /status (ordinals#3127)

* Add optional HTTP authentication for server (ordinals#3131)

* Make wallet async (ordinals#3142)

* Add `/r/inscription` endpoint for getting inscription details (ordinals#2628)

* Use `image-rendering: auto` for AVIF inscriptions (ordinals#3148)

* Make `ord env` more user friendly (ordinals#3153)

* Move JSON structs into api module (ordinals#3167)

* Display parent above metadata on /inscription (ordinals#3160)

* Represent rune IDs as `BLOCK:TX` (ordinals#3165)

* Add parent preview to inscription page (ordinals#3163)

* Update inscription sat documentation (ordinals#3114)

* Fix lints (ordinals#3124)

* Remove unnecessary lifetime from Formatter (ordinals#3178)

* Print PSBT for dry run inscribe (ordinals#3116)

* Update docs to reflect wallet changes (ordinals#3179)

* Improve configuration (ordinals#3156)

* Document `ord env` (ordinals#3180)

* Install openssl in docker image (ordinals#3181)

* Release 0.16.0-rc0 (ordinals#3182)

* Refactor test server to use arguments (ordinals#3183)

* Update ordinals crate (ordinals#3184)

* Release 0.16.0-rc1 (ordinals#3185)

* Allow configuring interval between commits to index (ordinals#3186)

* Credit contributors in changelog (ordinals#3187)

* Make deploys noninteractive (ordinals#3189)

* Make nop and burn tags one byte (ordinals#3207)

* Encode claims as tag (ordinals#3206)

* Add content proxy (ordinals#3216)

* Add reinscribe option to batch file (ordinals#3220)

* Check for duplicate satpoints in `satpoints` mode (ordinals#3221)

* Overhaul settings (ordinals#3188)

* Rename index_envelopes to index_inscriptions (ordinals#3233)

* Test that runes can be minted with no edict (ordinals#3231)

* Add libssl-dev to ubuntu install command (ordinals#3235)

* Enable indexing runes on mainnet (ordinals#3236)

* Document `ord wallet restore` (ordinals#3237)

* Load config from default data dir and configure `ord env ` using config (ordinals#3240)

* Document `ord env` commands (ordinals#3241)

* Fix list numbering in handbook (ordinals#3248)

* Display initial sync time on status page (ordinals#3250)

* Create tempdir in download-log recipe (ordinals#3242)

* Don't consider unconfirmed UTXOs as spent (ordinals#3255)

* Reserve inscription tag 15 (ordinals#3256)

* Rename genesis fee to inscription fee (ordinals#3257)

* Add more fields to /r/blockinfo (ordinals#3260)

* Add `id` inscription recursive JSON (ordinals#3258)

* Document recursive endpoint backwards compatibility guarantees (ordinals#3265)

* Release 0.16.0 (ordinals#3264)

- Bump version: 0.16.0-rc1 → 0.16.0
- Update changelog
- Update changelog contributor credits
- Update dependencies

* Bump ordinals version: 0.0.3 → 0.0.4 (ordinals#3267)

---------

Co-authored-by: Casey Rodarmor <[email protected]>
Co-authored-by: oxSaturn <[email protected]>
Co-authored-by: raph <[email protected]>
Co-authored-by: Robert Clarke <[email protected]>
Co-authored-by: Davirain <[email protected]>
Co-authored-by: Jeremy Rubin <[email protected]>
Co-authored-by: mj10021 <[email protected]>
Co-authored-by: zhiqiangxu <[email protected]>
Co-authored-by: 0xLugon <[email protected]>
Co-authored-by: Jerry the Martian <[email protected]>
Co-authored-by: HarveyV <[email protected]>
Co-authored-by: Michael Yu <[email protected]>
Co-authored-by: lifofifo <[email protected]>
Co-authored-by: Eloc <[email protected]>
Co-authored-by: Sitt Guruvanich <[email protected]>
Co-authored-by: bingryan <[email protected]>
Co-authored-by: Andrew <[email protected]>
Co-authored-by: Arik <[email protected]>
@GaloisField2718
Copy link

Hello @casey,
If we don't index spent sats how is it possible to know who is owning which satoshi?

Add --index-spent-sats option to keep sat index entries for spent outputs.

This is how the index originally worked, but it was too expensive, taking forever to build and swelling the size of the index to over 10 terabytes, if I recall correctly.

From your BitcoinTalk post: Unique serial number for every single satoshi, it sounds like index each spent satoshis.

This is untested, and is added for anyone who wants to try it out. It will likely require a crazily beefy machine to run, like 256GB+ of RAM, and 10s or 100s of terabytes of disk.

So the current indexer doesn't work concretely with this mechanism?

What index spent sats allow following this update?

@casey
Copy link
Collaborator Author

casey commented Jul 28, 2024

This option only pertains to whether or not prior locations of sats are stored, and does not affect whether or not the current location of each sat is stored.

@GaloisField2718
Copy link

Ok thanks, so it's keeping track of all trades of satoshis from day 1 on Bitcoin if I understand correctly?

@casey
Copy link
Collaborator Author

casey commented Jul 28, 2024

It's keeping a record of where sats used to be. Without --index-spent-sats, when a UTXO is spent, the database entry for that UTXO is deleted, to save space. With --index-spent-sats, database entries for spent UTXOs are retained, so you can query where sats used to be.

@GaloisField2718
Copy link

Ok very clear. Thank you very much for your precisions and your time!

@so7ow so7ow mentioned this pull request Aug 29, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raphjaph raphjaph raphjaph approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@casey @GaloisField2718 @raphjaph