Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store analysis to embedded K-V store #118

Closed
dundee opened this issue Jan 16, 2022 · 3 comments · Fixed by #297
Closed

Store analysis to embedded K-V store #118

dundee opened this issue Jan 16, 2022 · 3 comments · Fixed by #297
Labels
enhancement New feature or request

Comments

@dundee
Copy link
Owner

dundee commented Jan 16, 2022

Add a possibility to store the analysis into some fast embedded key-value store instead of memory. E.g. BadgerDB. The analysis would be of course slower a lot but it will use very little memory and the result could be opened again very fast.

@dundee dundee added the enhancement New feature or request label Jan 16, 2022
@dundee dundee added this to the 5.13 milestone Jan 16, 2022
@sss123next
Copy link

i can do tests with large datasets.

@dundee dundee removed this from the 5.13 milestone Sep 19, 2022
@lelandbr
Copy link

This is an awesome tool, thanks so much!

This db feature would be very useful for periodic scans of very large servers. duc is great for this, but it's way slower to scan than gdu on my server (as expected, based on your excellent benchmark data!). Once the scan is complete though, duc is much snappier to open/explore the result (stored in a Tokyocabinet database) than gdu (stored in a JSON), again, as expected.

This may also enable #150 - allowing users to view partially scanned dir trees if the scan aborts or is stopped. Restarting a scan from an aborted/stopped scan would also be amazing, but maybe tricky to do.

I would also find it useful to be able to merge/join two databases. Sometimes I find that the best way to reduce resource usage and scan time is to just take /large_dir/ and instead of scanning it all in one go, scan /large_dir/subdir1/ and then /large_dir/subdir2/ separately. But then those two subdirs have to be explored separately. If those two resulting databases could be combined (so that you can just interactively browse/large_dir/ all at once), that would be a really cool feature.

@dundee
Copy link
Owner Author

dundee commented Feb 12, 2024

First implementation released in https://github.com/dundee/gdu/releases/tag/v5.26.0
Please try it. All feedback appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants