Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improvements to Key enumeration of store.go #53

Open
3 tasks
kerneltime opened this issue Nov 26, 2018 · 4 comments
Open
3 tasks

Improvements to Key enumeration of store.go #53

kerneltime opened this issue Nov 26, 2018 · 4 comments
Assignees
@kerneltime @fredbi
Labels
core Issues related to datamon core P1 This is next and will move to P0 once planned. scalability

Comments

@kerneltime
Copy link
Contributor

kerneltime commented Nov 26, 2018 •
edited
Loading

The localfs implementation needs to

  • More efficiently walk the space for localfs (faster is better)

  • The API should allow for pagination to walk an arbitrarily large number of files.

  • Support for prefix key enumeration
@kerneltime kerneltime added the P1 This is next and will move to P0 once planned. label Nov 26, 2018
@kerneltime kerneltime added this to the General Availability milestone Nov 26, 2018
@kerneltime kerneltime changed the title Improvements to local fs walk of tree Improvements to Key enumeration of store.go Nov 26, 2018
@kerneltime
Copy link
Contributor Author

ref: https://github.com/karrick/godirwalk

@kerneltime kerneltime added P0 Highest priority. Current planned for work. and removed P1 This is next and will move to P0 once planned. labels Jan 15, 2019
@chandresh-pancholi
Copy link
Contributor

chandresh-pancholi commented Jan 21, 2019 •
edited
Loading

GCS

  • The API should allow for pagination to walk an arbitrarily large number of files.

  • Support for prefix key enumeration

  • Unit test

PR #68

@kerneltime kerneltime removed this from the General Availability milestone May 10, 2019
@kerneltime kerneltime added core Issues related to datamon core P1 This is next and will move to P0 once planned. and removed P0 Highest priority. Current planned for work. labels May 10, 2019
@fredbi
Copy link
Contributor

fredbi commented Feb 28, 2020

localfs case addressed by #404

We might still worry about the overall efficiency of key walking on a localfs. I don't see any better solution for now.

@fredbi fredbi mentioned this issue Feb 28, 2020
Merged
@fredbi
Copy link
Contributor

fredbi commented Mar 6, 2020

TODO: in core, replace calls to store.Keys by iterators.
Plus: should deprecate Keys in storage.Store

@fredbi fredbi self-assigned this Mar 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kerneltime kerneltime

@fredbi fredbi

Labels
core Issues related to datamon core P1 This is next and will move to P0 once planned. scalability
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kerneltime @chandresh-pancholi @fredbi