Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Umbrella issue for RW coordination for object storage. #1528

Closed
7 of 9 tasks
bwplotka opened this issue Sep 16, 2019 · 13 comments
Closed
7 of 9 tasks

Umbrella issue for RW coordination for object storage. #1528

bwplotka opened this issue Sep 16, 2019 · 13 comments

Comments

@bwplotka
Copy link
Member

bwplotka commented Sep 16, 2019

Agreed proposal: https://thanos.io/proposals/201901-read-write-operations-bucket.md/

Diagram

Action Items:

  • Ensure "consistency delay" (aka syncDelay) is implemented on all components (handling fresh blocks) with the reasonable minimum value
  • Ensure Store gateway handles toDelete blocks (detecting those and unloading before they are removed)
  • Implement handling of HealthyOverlapped block based on source ULIDs from meta file.
  • Delete should start from deleting meta.json (done)
  • Add ScheduleDelete that will delete adding deletion-mark.json
  • Add deleteDelay support for compactor on apply on toDelete blocks.
  • [Low] Save successful compactor block uploads to persistent file, same as shipper to avoid unnecessary duplicated compactions/downsamplings
  • [Low] Add ignore_delay parameter that will ignore syncDelay for sudden repairs.
  • [Low] Ignore HealthyOverlapped to reduce resource consumption for store gateway.
@bwplotka
Copy link
Member Author

cc @krasi-georgiev @mjudeikis

@jojohappy
Copy link
Member

@bwplotka I'm interested to implement the proposal, could I start to do that?

@bwplotka
Copy link
Member Author

Anyone is welcome to help. cc @abursavich as we talked about this today at KubeCon. Feel free to read the proposal: https://thanos.io/proposals/201901-read-write-operations-bucket.md/ It should be quite explanatory, even though a bit long. This issue description contains all tasks related to this. If you wish to start working on it I think we could join forces and finish it quickly, happy to jump on it finally as well.

@bwplotka
Copy link
Member Author

Fixes: #564

@palash25
Copy link

Hi @bwplotka I am also interested in the project. Since there are only two days left in the application period how are you going to select the mentee for this? Does one need to have previous contributions to thanos or a proposal on how to implement the project (like we do for GSoC) or both?

I am quite comfortable with Go with almost 2 years of experience writing it but distributed systems is something I have only begun to scratch the surface of and would like to get more experience in it. This seems like a perfect project for it.

Here is what I have done so far

  • Built thanos locally
  • Explored it a bit using the Katacode tutorial
  • Went through the project proposal

Haven't been through the code yet, will do it in the next few hours.
I am looking for some guidance here on what to do next and proceed further with my application for community bridge any help would be appreciated. Thanks

@bwplotka
Copy link
Member Author

Hi, Thanks for this @Masquerade0097 @palash25 ((:

However, I am not the one who decides. The full project list will be finalized on Monday: https://github.com/cncf/soc/blob/master/communitybridge/README.md#timeline

The CNCF will select projects and once the project will be selected, you will see it on the website and you will be able to apply as a mentee. Also cc @GiedriusS

Have a nice weekend. (:

@palash25
Copy link

thanks

@bwplotka
Copy link
Member Author

@palash25 @Masquerade0097 please apply for this as described here: https://docs.linuxfoundation.org/display/DOCS/Mentees

@palash25
Copy link

I did apply through the community bridge site @bwplotka

@bwplotka
Copy link
Member Author

bwplotka commented Feb 7, 2020

@khyatisoneji our Community Bridge student is on it (:

@stale
Copy link

stale bot commented Mar 8, 2020

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

@stale stale bot added the stale label Mar 8, 2020
@bwplotka
Copy link
Member Author

bwplotka commented Mar 8, 2020 via email

@stale stale bot removed the stale label Mar 8, 2020
@bwplotka
Copy link
Member Author

This should be now done, thanks to @khyatisoneji #2136

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants