Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OutputSweeper utility persisting and sweeping spendable outputs #2825

Merged

Conversation

tnull
Copy link
Contributor

@tnull tnull commented Jan 12, 2024

We add a OutputSweeper utility that allows to track the state of spendable output descriptors as emitted by Event::SpendableOutputs.

To this end, the OutputSweeper persists the necessary information in our KVStore and regularly tries to sweep the spendable outputs, removing them after reaching threshold confirmations, i.e., ANTI_REORG_DELAY.

This is a generalized version of the module previously added to LDK Node: lightningdevkit/ldk-node#152.

TODOs:

  • Add test coverage (likely in BackgroundProcessor tests)

Copy link

coderabbitai bot commented Jan 12, 2024

Note

Reviews Paused

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Walkthrough

The codebase introduces an OutputSweeper utility designed to manage spendable outputs within a key-value store. This module ensures that outputs are persistently tracked and attempts to sweep them by rebroadcasting spending transactions as necessary. It also handles transaction confirmations, both unconfirmed and confirmed, enhancing the robustness of transaction management.

Changes

File Path Change Summary
.../src/util/mod.rs Added a new module sweep.
.../src/util/persist.rs Added constants for output persistence namespaces.
.../src/util/sweep.rs Introduces the OutputSweeper utility for managing spendable outputs.

Poem

🐇 CodeRabbit hopped along, 🌟
Adding sweeps to code's sweet song. 🎶
Outputs tracked, no coin unturned, 🪙
In crypto's fire, wisdom burned. 🔥


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 4b70921 and 0df5bcc.
Files selected for processing (3)
  • lightning/src/util/mod.rs (1 hunks)
  • lightning/src/util/persist.rs (1 hunks)
  • lightning/src/util/sweep.rs (1 hunks)
Additional comments: 15
lightning/src/util/mod.rs (1)
  • 24-24: The addition of the sweep module is noted and appears to be correctly placed within the util module structure.
lightning/src/util/sweep.rs (13)
  • 7-10: The module documentation for OutputSweeper clearly explains its purpose and functionality.

  • 11-23: The imports and dependencies for the OutputSweeper are correctly listed and seem appropriate for the functionality described.

  • 30-31: The constant DEFAULT_REGENERATE_SPEND_THRESHOLD is well-documented and its purpose is clear.

  • 33-68: The TrackedSpendableOutput struct is well-defined with clear documentation for each field. It encapsulates the state of a spendable output.

  • 70-108: The methods to_watched_output and is_spent_in within TrackedSpendableOutput are logically sound and well-implemented.

  • 110-119: The implementation of impl_writeable_tlv_based! for TrackedSpendableOutput is correct and ensures that the struct can be serialized and deserialized.

  • 121-288: The OutputSweeper struct and its constructor methods are well-documented and structured. The use of generics and trait bounds is appropriate, and the code is clean and maintainable.

  • 291-341: The method track_spendable_outputs is logically sound, with clear handling of the exclude_static_outputs flag. Error handling with logging is also correctly implemented.

  • 343-346: The method tracked_spendable_outputs provides a simple and effective way to retrieve the current state of tracked outputs.

  • 348-428: The method rebroadcast_if_necessary contains logic to rebroadcast transactions as needed. The logic is sound, but it's important to ensure that the error handling in the persistence calls does not inadvertently swallow errors that should cause a failure in the calling code.

Verification successful

The persist_info method in the OutputSweeper utility logs errors encountered during the persistence process and then returns those errors to the caller. This means that the method does not swallow errors; instead, it allows the calling code to handle the error as deemed appropriate. This addresses the concern raised in the review comment regarding error handling.

Scripts Executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Verify that the error handling in the persistence calls of rebroadcast_if_necessary does not swallow critical errors.
# This script is a placeholder for the actual verification process, which would involve a deeper review of the error handling strategy.
echo "Review the error handling strategy for persistence calls in rebroadcast_if_necessary."

Length of output: 180



Script:

#!/bin/bash
# Examine the implementation of `self.persist_info` to understand how errors are handled.
ast-grep --lang rust --pattern 'impl OutputSweeper {
  $$$
  fn persist_info($_: &$_) -> $_ {
    $$$
  }
  $$$
}'

Length of output: 116



Script:

#!/bin/bash
# Use rg to search for the persist_info method within the OutputSweeper implementation.
rg "fn persist_info"

Length of output: 134



Script:

#!/bin/bash
# Extract the full implementation of the persist_info method from the sweep.rs file.
rg -A 10 "fn persist_info" lightning/src/util/sweep.rs

Length of output: 399



Script:

#!/bin/bash
# Extend the rg command to capture more lines following the persist_info method to see the complete error handling logic.
rg -A 20 "fn persist_info" lightning/src/util/sweep.rs

Length of output: 636

* 430-485: The method `prune_confirmed_outputs` and `persist_info` are well-implemented with appropriate error handling and logging.
  • 488-589: The implementations of the Listen and Confirm traits for OutputSweeper are correct and handle block connection/disconnection and transaction confirmation/unconfirmation appropriately.

  • 613-651: The function read_spendable_outputs and the helper id_to_hex_string are correctly implemented and handle errors appropriately.

lightning/src/util/persist.rs (1)
  • 68-77: The addition of the constants SPENDABLE_OUTPUT_INFO_PERSISTENCE_PRIMARY_NAMESPACE and SPENDABLE_OUTPUT_INFO_PERSISTENCE_SECONDARY_NAMESPACE is consistent with the existing pattern for defining namespaces for persistence. These constants are used to namespace the persistence of TrackedSpendableOutputs by the OutputSweeper. The primary namespace is set to "spendable_outputs", and the secondary namespace is an empty string, which is a valid and common pattern in this codebase.

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from 0df5bcc to da27965 Compare January 12, 2024 13:12
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 4b70921 and da27965.
Files selected for processing (3)
  • lightning/src/util/mod.rs (1 hunks)
  • lightning/src/util/persist.rs (1 hunks)
  • lightning/src/util/sweep.rs (1 hunks)
Files skipped from review as they are similar to previous changes (3)
  • lightning/src/util/mod.rs
  • lightning/src/util/persist.rs
  • lightning/src/util/sweep.rs

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from da27965 to ed9f8bb Compare January 12, 2024 13:15
@tnull
Copy link
Contributor Author

tnull commented Jan 12, 2024

@coderabbitai pause

@codecov-commenter
Copy link

codecov-commenter commented Jan 12, 2024

Codecov Report

Attention: Patch coverage is 57.33533% with 285 lines in your changes are missing coverage. Please review.

Project coverage is 89.18%. Comparing base (ac9a2c8) to head (be8c0d0).

Files Patch % Lines
lightning/src/util/sweep.rs 49.15% 255 Missing and 17 partials ⚠️
lightning-background-processor/src/lib.rs 92.47% 7 Missing ⚠️
lightning/src/chain/mod.rs 0.00% 4 Missing ⚠️
lightning/src/sign/mod.rs 94.28% 0 Missing and 2 partials ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2825      +/-   ##
==========================================
- Coverage   89.42%   89.18%   -0.25%     
==========================================
  Files         117      118       +1     
  Lines       96290    96920     +630     
  Branches    96290    96920     +630     
==========================================
+ Hits        86109    86437     +328     
- Misses       7962     8240     +278     
- Partials     2219     2243      +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch 3 times, most recently from dc0340e to c5052e2 Compare January 12, 2024 16:49
@tnull
Copy link
Contributor Author

tnull commented Jan 12, 2024

Added first basic testcase in the BackgroundProcessor tests. This could be expanded further, but I think it's ready for a first round of review now.

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch 4 times, most recently from 29b4a7a to 0626f34 Compare January 15, 2024 10:52
Copy link
Contributor

@jbesraa jbesraa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

and proposed to review club lightningdevkit/ldk-review-club#28

lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from 0626f34 to ca91946 Compare January 24, 2024 11:23
Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, I assume this is already in ldk-node so there's no use commenting on particular storage format?

lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
self.outputs.lock().unwrap().clone()
}

fn rebroadcast_if_necessary(&self) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make the broadcasting optional here so that users can use the utility to just spend when they need to spend? Would require just letting the user skip this and then refactoring the spend logic to take an extra (few) output(s) when called manually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure what you're suggesting. What would be the benefit of skipping the broadcast, and how would it then be different from KeyManager::spend_spendable_output? Given that broadcasting is always fallible without us getting feedback until we see a confirmation, what would be a scenario where rebroadcasting every block isn't desired? Also, what do you mean with 'extra few outputs'?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would it then be different from KeyManager::spend_spendable_output?

Mostly just that there would be a utility to handle the storage for you. I agree its not a big difference and certainly anyone managing their own storage can just do that.

Given that broadcasting is always fallible without us getting feedback until we see a confirmation, what would be a scenario where rebroadcasting every block isn't desired? Also, what do you mean with 'extra few outputs'?

I didn't mean not rebroadcasting, only that it'd be nice to let users delay broadcasting until they want to do some other onchain transaction and can just include the sweeping then.

}

if let Some(latest_broadcast_height) = output_info.latest_broadcast_height {
// Re-generate spending tx after regenerate_spend_threshold, rebroadcast
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we end up with some output that got claimed another way, won't this mean all our future spends are gonna get stuck?

Copy link
Contributor Author

@tnull tnull Feb 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intermittently yes, but we'll still pick up on any confirmed spends from which point on we wouldn't try to include the outputs in the spending transaction regeneration. (Note that in Confirm::transactions_confirmed here we don't match on the txid but check for any confirmed transaction whether it spends one of our tracked outputs).

We'd then also prune the 'stuck' outputs once the (potentially third-party) spend is sufficiently confirmed.

If that's not acceptable we could of course fall back to individually retry to spend & broadcast outputs that previously failed, however, I'd rather not go that route if it's not really necessary as generating one spend per output is really wasteful fee-wise.

lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
@tnull
Copy link
Contributor Author

tnull commented Feb 2, 2024

Basically LGTM, I assume this is already in ldk-node so there's no use commenting on particular storage format?

Yes, essentially. We could of course always find some migration steps if it would be really necessary though. Is there anything in particular that stands out to you?

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from ca91946 to f418234 Compare February 2, 2024 08:58
@tnull
Copy link
Contributor Author

tnull commented Feb 2, 2024

Rebased on main to resolve minor conflict.

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from f418234 to 3db8480 Compare February 2, 2024 09:14
Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs another rebase, sorry about that. Basically LGTM, I was just wondering if I should spend time trying to puzzle over a simpler (read: less stateful) design or if this is kinda fixed and we should just ship it as long as its good enough (it more than is).

lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
@tnull
Copy link
Contributor Author

tnull commented Feb 12, 2024

Needs another rebase, sorry about that. Basically LGTM, I was just wondering if I should spend time trying to puzzle over a simpler (read: less stateful) design or if this is kinda fixed and we should just ship it as long as its good enough (it more than is).

Alright. Actually I had originally started out with keeping more state and tracking in a proper STM, which had some benefits w.r.t. explicitness of transitions and would give the user access to complimentary information, but that was reduced during the review on LDK Node. If there's no major concern it's probably best to keep it as is (read: not unnecessarily break compat with LDK Node's sweeper).

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from 3db8480 to eff8db9 Compare February 12, 2024 11:44
@tnull
Copy link
Contributor Author

tnull commented Feb 12, 2024

Rebased on main to resolve minor conflict.

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch 2 times, most recently from f107d77 to dd09907 Compare February 12, 2024 13:28
@tnull tnull added this to the 0.0.122 milestone Feb 20, 2024
@jkczyz jkczyz self-requested a review February 28, 2024 00:42
) -> Result<Self, DecodeError> {
let (broadcaster, chain_data_source, kv_store, logger, spend_outputs_callback) = args;
let state = SweeperState::read(reader)?;
let best_block = state.best_block;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should enforce this instead by defining a trait for returning the BestBlock(or even add a best_block method to chain::Listen / chain::Confirm) such that our utilities query it rather than relying on the user providing it paired with the given type. Returning it as part of Readable doesn't prevent the user from doing the wrong thing.

lightning/src/util/sweep.rs Show resolved Hide resolved
lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
if i == num_blocks {
node.node.best_block_updated(&header, height);
node.chain_monitor.best_block_updated(&header, height);
node.sweeper.best_block_updated(&header, height);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we only testing this using lightning-background-processor because our functional tests utilities don't allow additional chain::Listen/chain::Confirm implementations?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one reason, but also as we require a 'real' KeysManager allowing to spend the outputs. Essentially we require most of the core components of a Lightning node, and our BP tests seems to be the only place where all come together?

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch 2 times, most recently from 986bfcd to 2c8fffc Compare April 16, 2024 12:52
@tnull
Copy link
Contributor Author

tnull commented Apr 16, 2024

Now updated to

  • Drop the spend_spendable_outputs callback fn in favor of using traits. To this end we now introduced OutputSpender (impl on (Phantom)KeysMananger) and ChangeDestinationSource traits as well as a new ConfirmationTarget::OutputSpendingFee (all naming up for discussion, ofc).
  • Allow delaying the spend generation via a enum SpendingDelay until a relative or absolute block height is reached. Also added some test coverage.
  • Address previous round of feedback.
  • Rebased to include rustfmt changes in sign before touching these files

@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch 3 times, most recently from 3ef6e21 to 4dfcfc0 Compare April 16, 2024 16:46
lightning/src/sign/mod.rs Outdated Show resolved Hide resolved
lightning/src/chain/chaininterface.rs Outdated Show resolved Hide resolved
lightning/src/chain/chaininterface.rs Outdated Show resolved Hide resolved
lightning/src/events/mod.rs Outdated Show resolved Hide resolved
lightning/src/util/sweep.rs Show resolved Hide resolved
lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
Comment on lines 720 to 721
state_lock.best_block = BestBlock::new(header.block_hash(), height);
self.prune_confirmed_outputs(&mut *state_lock);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not worth it, but could use the result of these to determine if persistence can be skipped. Or do we always need to persist when the best block is updated?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, yeah, previously I left this one alone as I assumed we'd want to persist the BestBlock. Whether we could do that lazily might be debatable. Note it would only safe a single persist call on each block connect if we don't have any outputs that are currently broadcasted, as otherwise we'd always update latest_broadcast_height anyways.

If we want to do it lazily, we might need to consider introducing an interval in which we force persisting the best block, as otherwise long-running nodes with low traffic might not persist the BestBlock for a long time, and we'd then force them to (unnecessarily) resync a lot of chain data on restart.

lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from 4dfcfc0 to e362dac Compare April 17, 2024 09:24
@tnull
Copy link
Contributor Author

tnull commented Apr 17, 2024

Addressed feedback and rebased to resolve minor conflicts.

lightning/src/util/sweep.rs Outdated Show resolved Hide resolved
/// Will be `None` if no `channel_id` was given to [`OutputSweeper::track_spendable_outputs`]
pub channel_id: Option<ChannelId>,
/// The current status of the output spend.
pub status: OutputSpendStatus,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, so I can't say I'm a huge fan of all the internal state being publicly exposed, it means we don't really do any validation at all of what the user tells us the state of an output is and all the debug assertions in this file are reachable if the user simply provides some garbage.

If we want to expose them all so that users can inspect using tracked_spendable_outputs maybe we should make from_outputs #[doc(hidden)] so that users are discouraged from passing invalid state that we can't verify?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, so I can't say I'm a huge fan of all the internal state being publicly exposed, it means we don't really do any validation at all of what the user tells us the state of an output is and all the debug assertions in this file are reachable if the user simply provides some garbage.

If we want to expose them all so that users can inspect using tracked_spendable_outputs maybe we should make from_outputs #[doc(hidden)] so that users are discouraged from passing invalid state that we can't verify?

I think we def. want to expose the TrackedSpendableOutputs to allow users insight into what state their funds are in. Currently (and otherwise going forward), the funds just disappear and reappear at some magic time in the user's on-chain wallet if everything goes according to plan. To make this more transparent, I think it's important to give full insight into what's happening (and even allow them to grab the Transaction and broadcast it out-of-band if something goes wrong). This is one reason why I previously was hesitant to make too many assumptions about what status a transaction is in, if we don't have explicit data on it.

That said, if you're worried users would feed us bogus data, we can probably drop from_outputs from the API now that we implemented ReadableArgs on OutputSweeper. Now added a fixup doing so.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said, if you're worried users would feed us bogus data, we can probably drop from_outputs from the API now that we implemented ReadableArgs on OutputSweeper. Now added a fixup doing so.

Don't you need it for ldk-node?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you need it for ldk-node?

By now I kind of gave up on the idea of reconstructing the new data types from the old persisted state and would probably just re-add the persisted descriptors without any prior history, which I think it should work fine as a one-time migration step and seems cleaner than having a hidden API call lying around here.

lightning/src/util/sweep.rs Show resolved Hide resolved
lightning/src/util/sweep.rs Show resolved Hide resolved
}

/// Returns whether the output is spent in the given transaction.
pub fn is_spent_in(&self, tx: &Transaction) -> bool {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for this to be pub?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For one it it may be a useful util to have. For example, I used it in the tests in BP. If you'd rather not have it pub, happy to either copy over the code or do a cfg(test)/cfg(not(test)) dance.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess? Just seems like a kind of random utility to expose.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess? Just seems like a kind of random utility to expose.

True, happy to go either way really. Really not sure what I prefer here (keeping it pub, copying code, adding cfg boilerplate). 🤷‍♂️

lightning/src/util/sweep.rs Show resolved Hide resolved
@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from e362dac to 71b3225 Compare April 18, 2024 09:33
@TheBlueMatt
Copy link
Collaborator

LGTM, feel free to squash.

.. we move `spend_spendable_outputs` to a new trait which we can easily
reuse in `OutputSweeper` later.
.. which users should implement on their on-chain wallet to allow us to
retrieve a new change destination script.
@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from 71b3225 to 9a24220 Compare April 18, 2024 14:09
@tnull
Copy link
Contributor Author

tnull commented Apr 18, 2024

LGTM, feel free to squash.

Squashed without further changes.

We add a `OutputSweeper` utility that allows to track the state of
spendable output descriptors as emitted by `Event::SpendableOutputs`.

To this end, the `OutputSweeper` persists the necessary information in
our `KVStore` and regularly tries to sweep the spendable outputs,
removing them after reaching threshold confirmations, i.e.,
`ANTI_REORG_DELAY`.
.. which enables users to batch output spends.
.. we simply check that the `OutputSweeper` generates a spending tx and
that the `TrackedSpendableOutput` is pruned once it reaches
`ANTI_REORG_DELAY`.
@tnull tnull force-pushed the 2024-01-upstream-output-sweeper branch from 9a24220 to be8c0d0 Compare April 18, 2024 14:20
@tnull
Copy link
Contributor Author

tnull commented Apr 18, 2024

Force pushed without any net changes moving the OutputSpendingFee docs to 071337a to make check_commits CI happy (as the OutputSweeper isn't available yet in the previous commit).

@TheBlueMatt TheBlueMatt merged commit fd23710 into lightningdevkit:main Apr 18, 2024
15 of 16 checks passed
k0k0ne pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
v0.0.123 - May 08, 2024 - "BOLT12 Dust Sweeping"

API Updates
===========

 * To reduce risk of force-closures and improve HTLC reliability the default
   dust exposure limit has been increased to
   `MaxDustHTLCExposure::FeeRateMultiplier(10_000)`. Users with existing
   channels might want to consider using
   `ChannelManager::update_channel_config` to apply the new default (lightningdevkit#3045).
 * `ChainMonitor::archive_fully_resolved_channel_monitors` is now provided to
   remove from memory `ChannelMonitor`s that have been fully resolved on-chain
   and are now not needed. It uses the new `Persist::archive_persisted_channel`
   to inform the storage layer that such a monitor should be archived (lightningdevkit#2964).
 * An `OutputSweeper` is now provided which will automatically sweep
   `SpendableOutputDescriptor`s, retrying until the sweep confirms (lightningdevkit#2825).
 * After initiating an outbound channel, a peer disconnection no longer results
   in immediate channel closure. Rather, if the peer is reconnected before the
   channel times out LDK will automatically retry opening it (lightningdevkit#2725).
 * `PaymentPurpose` now has separate variants for BOLT12 payments, which
   include fields from the `invoice_request` as well as the `OfferId` (lightningdevkit#2970).
 * `ChannelDetails` now includes a list of in-flight HTLCs (lightningdevkit#2442).
 * `Event::PaymentForwarded` now includes `skimmed_fee_msat` (lightningdevkit#2858).
 * The `hashbrown` dependency has been upgraded and the use of `ahash` as the
   no-std hash table hash function has been removed. As a consequence, LDK's
   `Hash{Map,Set}`s no longer feature several constructors when LDK is built
   with no-std; see the `util::hash_tables` module instead. On platforms that
   `getrandom` supports, setting the `possiblyrandom/getrandom` feature flag
   will ensure hash tables are resistant to HashDoS attacks, though the
   `possiblyrandom` crate should detect most common platforms (lightningdevkit#2810, lightningdevkit#2891).
 * `ChannelMonitor`-originated requests to the `ChannelSigner` can now fail and
   be retried using `ChannelMonitor::signer_unblocked` (lightningdevkit#2816).
 * `SpendableOutputDescriptor::to_psbt_input` now includes the `witness_script`
   where available as well as new proprietary data which can be used to
   re-derive some spending keys from the base key (lightningdevkit#2761, lightningdevkit#3004).
 * `OutPoint::to_channel_id` has been removed in favor of
   `ChannelId::v1_from_funding_outpoint` in preparation for v2 channels with a
   different `ChannelId` derivation scheme (lightningdevkit#2797).
 * `PeerManager::get_peer_node_ids` has been replaced with `list_peers` and
   `peer_by_node_id`, which provide more details (lightningdevkit#2905).
 * `Bolt11Invoice::get_payee_pub_key` is now provided (lightningdevkit#2909).
 * `Default[Message]Router` now take an `entropy_source` argument (lightningdevkit#2847).
 * `ClosureReason::HTLCsTimedOut` has been separated out from
   `ClosureReason::HolderForceClosed` as it is the most common case (lightningdevkit#2887).
 * `ClosureReason::CooperativeClosure` is now split into
   `{Counterparty,Locally}Initiated` variants (lightningdevkit#2863).
 * `Event::ChannelPending::channel_type` is now provided (lightningdevkit#2872).
 * `PaymentForwarded::{prev,next}_user_channel_id` are now provided (lightningdevkit#2924).
 * Channel init messages have been refactored towards V2 channels (lightningdevkit#2871).
 * `BumpTransactionEvent` now contains the channel and counterparty (lightningdevkit#2873).
 * `util::scid_utils` is now public, with some trivial utilities to examine
   short channel ids (lightningdevkit#2694).
 * `DirectedChannelInfo::{source,target}` are now public (lightningdevkit#2870).
 * Bounds in `lightning-background-processor` were simplified by using
   `AChannelManager` (lightningdevkit#2963).
 * The `Persist` impl for `KVStore` no longer requires `Sized`, allowing for
   the use of `dyn KVStore` as `Persist` (lightningdevkit#2883, lightningdevkit#2976).
 * `From<PaymentPreimage>` is now implemented for `PaymentHash` (lightningdevkit#2918).
 * `NodeId::from_slice` is now provided (lightningdevkit#2942).
 * `ChannelManager` deserialization may now fail with `DangerousValue` when
    LDK's persistence API was violated (lightningdevkit#2974).

Bug Fixes
=========

 * Excess fees on counterparty commitment transactions are now included in the
   dust exposure calculation. This lines behavior up with some cases where
   transaction fees can be burnt, making them effectively dust exposure (lightningdevkit#3045).
 * `Future`s used as an `std::...::Future` could grow in size unbounded if it
   was never woken. For those not using async persistence and using the async
   `lightning-background-processor`, this could cause a memory leak in the
   `ChainMonitor` (lightningdevkit#2894).
 * Inbound channel requests that fail in
   `ChannelManager::accept_inbound_channel` would previously have stalled from
   the peer's perspective as no `error` message was sent (lightningdevkit#2953).
 * Blinded path construction has been tuned to select paths more likely to
   succeed, improving BOLT12 payment reliability (lightningdevkit#2911, lightningdevkit#2912).
 * After a reorg, `lightning-transaction-sync` could have failed to follow a
   transaction that LDK needed information about (lightningdevkit#2946).
 * `RecipientOnionFields`' `custom_tlvs` are now propagated to recipients when
   paying with blinded paths (lightningdevkit#2975).
 * `Event::ChannelClosed` is now properly generated and peers are properly
   notified for all channels that as a part of a batch channel open fail to be
   funded (lightningdevkit#3029).
 * In cases where user event processing is substantially delayed such that we
   complete multiple round-trips with our peers before a `PaymentSent` event is
   handled and then restart without persisting the `ChannelManager` after having
   persisted a `ChannelMonitor[Update]`, on startup we may have `Err`d trying to
   deserialize the `ChannelManager` (lightningdevkit#3021).
 * If a peer has relatively high latency, `PeerManager` may have failed to
   establish a connection (lightningdevkit#2993).
 * `ChannelUpdate` messages broadcasted for our own channel closures are now
   slightly more robust (lightningdevkit#2731).
 * Deserializing malformed BOLT11 invoices may have resulted in an integer
   overflow panic in debug builds (lightningdevkit#3032).
 * In exceedingly rare cases (no cases of this are known), LDK may have created
   an invalid serialization for a `ChannelManager` (lightningdevkit#2998).
 * Message processing latency handling BOLT12 payments has been reduced (lightningdevkit#2881).
 * Latency in processing `Event::SpendableOutputs` may be reduced (lightningdevkit#3033).

Node Compatibility
==================

 * LDK's blinded paths were inconsistent with other implementations in several
   ways, which have been addressed (lightningdevkit#2856, lightningdevkit#2936, lightningdevkit#2945).
 * LDK's messaging blinded paths now support the latest features which some
   nodes may begin relying on soon (lightningdevkit#2961).
 * LDK's BOLT12 structs have been updated to support some last-minute changes to
   the spec (lightningdevkit#3017, lightningdevkit#3018).
 * CLN v24.02 requires the `gossip_queries` feature for all peers, however LDK
   by default does not set it for those not using a `P2PGossipSync` (e.g. those
   using RGS). This change was reverted in CLN v24.02.2 however for now LDK
   always sets the `gossip_queries` feature. This change is expected to be
   reverted in a future LDK release (lightningdevkit#2959).

Security
========
0.0.123 fixes a denial-of-service vulnerability which we believe to be reachable
from untrusted input when parsing invalid BOLT11 invoices containing non-ASCII
characters.
 * BOLT11 invoices with non-ASCII characters in the human-readable-part may
   cause an out-of-bounds read attempt leading to a panic (lightningdevkit#3054). Note that all
   BOLT11 invoices containing non-ASCII characters are invalid.

In total, this release features 150 files changed, 19307 insertions, 6306
deletions in 360 commits since 0.0.121 from 17 authors, in alphabetical order:

 * Arik Sosman
 * Duncan Dean
 * Elias Rohrer
 * Evan Feenstra
 * Jeffrey Czyz
 * Keyue Bao
 * Matt Corallo
 * Orbital
 * Sergi Delgado Segura
 * Valentine Wallace
 * Willem Van Lint
 * Wilmer Paulino
 * benthecarman
 * jbesraa
 * olegkubrakov
 * optout
 * shaavan
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants