Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(share/ipld): Remove proofs from leaked ctx #2574

Merged

Conversation

walldiss
Copy link
Member

@walldiss walldiss commented Aug 16, 2023

Inside tee-getter ProofAdder is added to ctx to collect proofs and allow eds store to read them, without extra recompute. Ctx provided to store.Put method, will be used for register shard, that in some rare cases takes long time and returns, before result returned from operation. The provided context is still leaked to async operation, and it prevents golang GC from collecting memory allocated for ProofAdder, because there is still possibility for code to reach it.

Solution
Dagstore register shard operation does not need to access to proofs, so the associated memory could be reclaimed right after proofs are extracted from ctx. For convenience it will happen after put operation

Solves first point of #2582

@walldiss walldiss added area:storage kind:fix Attached to bug-fixing PRs labels Aug 16, 2023
@walldiss walldiss self-assigned this Aug 16, 2023
@codecov-commenter
Copy link

Codecov Report

Merging #2574 (a0d4a7e) into main (003c2c4) will increase coverage by 0.05%.
Report is 1 commits behind head on main.
The diff coverage is 43.47%.

@@            Coverage Diff             @@
##             main    #2574      +/-   ##
==========================================
+ Coverage   51.01%   51.07%   +0.05%     
==========================================
  Files         158      158              
  Lines       10398    10428      +30     
==========================================
+ Hits         5305     5326      +21     
- Misses       4627     4636       +9     
  Partials      466      466              
Files Changed Coverage Δ
share/ipld/nmt_adder.go 45.71% <0.00%> (-3.78%) ⬇️
share/eds/eds.go 63.73% <40.00%> (-0.16%) ⬇️
core/exchange.go 35.84% <50.00%> (+0.55%) ⬆️
core/listener.go 53.50% <100.00%> (-1.85%) ⬇️
share/getters/tee.go 42.50% <100.00%> (+4.66%) ⬆️

... and 4 files with indirect coverage changes

@walldiss walldiss marked this pull request as ready for review August 17, 2023 10:52
Wondertan
Wondertan previously approved these changes Aug 17, 2023
Copy link
Member

@Wondertan Wondertan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you've tested this and seen stable RAM usage on Graphana

share/ipld/nmt_adder.go Show resolved Hide resolved
@Wondertan
Copy link
Member

The fact that proofs are leaking is a hint that the context is leaking. Because if the context is cleared up, the proofs would be cleared up as well. WDYT?

@walldiss
Copy link
Member Author

walldiss commented Aug 18, 2023

The fact that proofs are leaking is a hint that the context is leaking. Because if the context is cleared up, the proofs would be cleared up as well. WDYT?

I agree, Thats why PR is named for issue of leaked ctx, but not just proofs "Remove proofs from leaked ctx". Provided solution is a hotfix for latest release to prevent leak. Proper investigation of leaked ctx will take some time, since it is passed into many complex components, like ipld, bitswap, dagstore, badger. I'll create an issue to investigate where the ctx is leaked.

#2582

share/ipld/nmt_adder.go Outdated Show resolved Hide resolved
@Wondertan Wondertan enabled auto-merge (squash) August 18, 2023 14:17
@Wondertan Wondertan merged commit 04e2928 into celestiaorg:main Aug 18, 2023
12 of 14 checks passed
renaynay pushed a commit to renaynay/celestia-node that referenced this pull request Aug 23, 2023
walldiss added a commit to walldiss/celestia-node that referenced this pull request Sep 22, 2023
walldiss added a commit that referenced this pull request Sep 22, 2023
walldiss added a commit to walldiss/celestia-node that referenced this pull request Sep 22, 2023
walldiss added a commit to walldiss/celestia-node that referenced this pull request Sep 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:storage kind:fix Attached to bug-fixing PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants