Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate the role of chunk size streaming. #59

Open
kerneltime opened this issue Nov 30, 2018 · 1 comment
Open

Investigate the role of chunk size streaming. #59

kerneltime opened this issue Nov 30, 2018 · 1 comment
Assignees
Labels
backend issues related to the backend services datamon depends on P2 We know the issue but no plans in place for implementation. performance Performance related issues spike Design, investigation tasks or tasks that does not have clear deliverable

Comments

@kerneltime
Copy link
Contributor

Currently the upload to backend of CAFS waits on the buffer == leaf size being read in
Look into if sending the data to the backend without waiting for the buffer to be read in. The eventual key can be copied over by S3 (unclear if this will be faster).
Alternatively the chunk size can be played with.

@kerneltime kerneltime self-assigned this Nov 30, 2018
@kerneltime kerneltime added the P2 We know the issue but no plans in place for implementation. label Nov 30, 2018
@kerneltime kerneltime added this to the Beta milestone Nov 30, 2018
@kerneltime kerneltime removed this from the Beta milestone May 10, 2019
@kerneltime kerneltime added spike Design, investigation tasks or tasks that does not have clear deliverable backend issues related to the backend services datamon depends on labels May 10, 2019
@fredbi fredbi added the performance Performance related issues label Mar 5, 2020
@fredbi
Copy link
Contributor

fredbi commented Mar 6, 2020

@kerneltime looks like the current situation is as follows:

  • reads and writes are parallelized in chunks the size of a leaf
  • we do not split further down than the leaf size

I assume this is acceptable and that we could close this one, unless there is still something I missed
Please advise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend issues related to the backend services datamon depends on P2 We know the issue but no plans in place for implementation. performance Performance related issues spike Design, investigation tasks or tasks that does not have clear deliverable
Development

No branches or pull requests

2 participants