Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

esplora: fix incorrect gap limit check in blocking client #1225

Merged
merged 4 commits into from
Nov 27, 2023

Conversation

darosior
Copy link
Contributor

The gap limit was checked such as if the last_index was not None but the last_active_index was, we'd consider having reached it. But the last_index is never None for this check. This effectively made it so the gap limit was always 1: if the first address isn't used last_active_index will be None and we'd return immediately.

Fix this by avoiding error-prone Option comparisons and correctly checking we've reached the gap limit before breaking out of the loop.

@darosior
Copy link
Contributor Author

It's a bit surprising this bug wouldn't be caught by any test. I'll look into adding one.

@tnull
Copy link
Contributor

tnull commented Nov 20, 2023

From a real quick comparison in looks like the changed code is part of async_ext.rs. Does the async variant therefore also suffer from the same issue and needs to be fixed?

@darosior
Copy link
Contributor Author

Does the async variant therefore also suffer from the same issue and needs to be fixed?

Probably, but to be honest i'm pretty limited in time at the moment so i'm only writing a unit test to demonstrate the bug for the blocking client. I'd prefer if another contributor took care of fixing the async client.

@darosior darosior force-pushed the 2311_fix_explora_gap_limit branch from 65e1d36 to d5ecfd3 Compare November 20, 2023 10:55
@darosior
Copy link
Contributor Author

Pushed a unit test exercising this logic as a separate commit, so it can be cherry-picked on master to demonstrate the bug.

If somebody wants to take care of fixing the async client they can reuse the same logic from this test, i think.

@darosior
Copy link
Contributor Author

Ok so actually porting the fix and the test to the async client was actually trivial, so i just did it here. The same goes for the async unit test: it can be cherry-picked on master to demonstrate the bug.

@notmandatory notmandatory added the bug Something isn't working label Nov 20, 2023
@notmandatory notmandatory added this to the 1.0.0-alpha.3 milestone Nov 20, 2023
@LLFourn
Copy link
Contributor

LLFourn commented Nov 21, 2023

It's a bit surprising this bug wouldn't be caught by any test. I'll look into adding one.

I can see you've done this thanks. I think we are in the middle of setting up a proper test environment for this crate in #1171. I think you can leave @evanlinjin and @LagginTimes to polish of this PR after that.

Copy link
Member

@danielabrozzoni danielabrozzoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not very knowledgeable on the esplora code, I have a few questions

crates/esplora/tests/async_ext.rs Outdated Show resolved Hide resolved
crates/esplora/src/async_ext.rs Outdated Show resolved Hide resolved
Copy link
Member

@evanlinjin evanlinjin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for spotting this major bug and quickly implementing the fix with tests to go with. The changes here are strictly better than what we had before.

However, stop_gap is now implemented slightly incorrectly. If you do not have time to make these changes, I could do it for you!

crates/esplora/src/blocking_ext.rs Outdated Show resolved Hide resolved
crates/esplora/tests/blocking_ext.rs Outdated Show resolved Hide resolved
@evanlinjin
Copy link
Member

evanlinjin commented Nov 24, 2023

@danielabrozzoni sorry I didn't see your review before posting mine. Guess we are talking about the same thing haha.

Okay I think we both agree that stop_gap is not implemented correctly, but I think my suggestion is correcter.

The gap limit was checked such as if the last_index was not None but the
last_active_index was, we'd consider having reached it. But the
last_index is never None for this check. This effectively made it so the
gap limit was always 1: if the first address isn't used
last_active_index will be None and we'd return immediately.

Fix this by avoiding error-prone Option comparisons and correctly
checking we've reached the gap limit before breaking out of the loop.
@darosior darosior force-pushed the 2311_fix_explora_gap_limit branch from 9e1abe1 to 43aed38 Compare November 24, 2023 11:22
@darosior
Copy link
Contributor Author

Ok, just pushed a version which fixes the off-by-one when the last active index is None. Thanks both for chiming in!

@darosior
Copy link
Contributor Author

Ok the discussion about what the gap limit should be is now happening in two threads. Let's centralize it here. I'm going to close both threads.

First of all let me point out how this PR does not change existing behaviour for the case where there is a last active index. We might want to change that, but maybe let's keep this for another PR: one bug at a time heh?

That said, whether it happens here or in a follow-up, Daniela raises a good point. If we define "gap limit" as "the number of inactive derivation indexes before we stop" then the current implementation is wrong: as demonstrated by the newly introduced test with a gap limit of 3 we scan up to the 4th derivation index.

Copy link
Member

@danielabrozzoni danielabrozzoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all let me point out how this PR does not change existing behaviour for the case where there is a last active index. We might want to change that, but maybe let's keep this for another PR: one bug at a time heh?

Fair enough! I've opened #1227 for discussion about this - even if the current definition of stop_gap is correct, it should be clearly documented :)

ACK 43aed38

@darosior
Copy link
Contributor Author

I've opened #1127 for discussion

#1227

Copy link
Member

@evanlinjin evanlinjin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 43aed38

@evanlinjin evanlinjin merged commit 55b680c into bitcoindevkit:master Nov 27, 2023
12 checks passed
@notmandatory notmandatory mentioned this pull request Jan 3, 2024
12 tasks
@darosior darosior deleted the 2311_fix_explora_gap_limit branch September 2, 2024 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

6 participants