Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Epic: [Azure] production ready Azure blob storage support #5567

Closed
13 of 17 tasks
arpad-m opened this issue Oct 16, 2023 · 3 comments
Closed
13 of 17 tasks

Epic: [Azure] production ready Azure blob storage support #5567

arpad-m opened this issue Oct 16, 2023 · 3 comments
Assignees
@arpad-m
Labels
c/storage/pageserver Component: storage: pageserver c/storage/safekeeper Component: storage: safekeeper f/azure t/Epic Issue type: Epic team/azure

Comments

@arpad-m
Copy link
Member

arpad-m commented Oct 16, 2023 •
edited by fcdm
Loading

PR #5546 has added a preview-level azure blob storage support but left some key areas in a state that could see improvements.
This epic tracks the follow-up work needed to make azure blob storage support production ready.

Tasks
Preview Give feedback
  1. Use streaming APIs for Azure uploads/downloads #5563
    c/storage/pageserver c/storage/safekeeper
    arpad-m
  4. Metrics support for azure blob storage #5590
    c/storage/pageserver c/storage/safekeeper f/azure
  5. Support Active Directory for Azure authentication #5566
    c/storage
  6. test test_real_azure.rs in CI #5565
    c/storage
    bayandin
  8. Ensure metadata atomicity in Azure Blob Storage #5571
    c/storage/pageserver c/storage/safekeeper
  9. Azure blobs: Simplify error conversion by addition of to_download_error #5575
  10. Cleanup in azure_upload_download_works test #5636
  11. Update azure SDK to 0.18 and use open range support #6103
  12. Add copy operation tests and implement them for azure blobs #6362
  13. Increase Azure parallelism limit to 100 #7597
  14. Perform retries on azure bulk deletion #7964
  15. Epic: Support Azure Blob storage in the scrubber #7547
    c/storage c/storage/scrubber t/Epic
    arpad-m
  16. Implement time_travel_recover for Azure blob storage #7546
    a/tech_debt c/storage
    arpad-m
@arpad-m arpad-m added t/Epic Issue type: Epic c/storage Component: storage labels Oct 16, 2023
@arpad-m arpad-m mentioned this issue Oct 16, 2023
Merged
@arpad-m arpad-m self-assigned this Oct 16, 2023
@arpad-m arpad-m added c/storage/pageserver Component: storage: pageserver c/storage/safekeeper Component: storage: safekeeper and removed c/storage Component: storage labels Oct 16, 2023
@stepashka
Copy link
Member

@arpad-m , does this replace https://github.com/neondatabase/cloud/issues/7204 ?

@arpad-m
Copy link
Member Author

arpad-m commented Oct 16, 2023

@stepashka there is also #5562 which is soon to be closed. I think with that in combination with this issue, we can say https://github.com/neondatabase/cloud/issues/7204 is replaced. Sorry I've not seen it!

arpad-m added a commit that referenced this issue Oct 16, 2023
@arpad-m
Azure blob storage support (#5546)
e09d5ad 
Adds prototype-level support for [Azure blob storage](https://azure.microsoft.com/en-us/products/storage/blobs). Some corners were cut, see the TODOs and the followup issue #5567 for details.

Steps to try it out:

* Create a storage account with block blobs (this is a per-storage
account setting).
* Create a container inside that storage account.
* Set the appropriate env vars: `AZURE_STORAGE_ACCOUNT,
AZURE_STORAGE_ACCESS_KEY, REMOTE_STORAGE_AZURE_CONTAINER,
REMOTE_STORAGE_AZURE_REGION`
* Set the env var `ENABLE_REAL_AZURE_REMOTE_STORAGE=y` and run `cargo
test -p remote_storage azure`

Fixes  #5562
This was referenced Oct 19, 2023
Closed
Closed
Closed
Closed
@arpad-m arpad-m mentioned this issue Nov 2, 2023
Open
This was referenced Dec 7, 2023
Merged
Merged
arpad-m added a commit that referenced this issue Dec 12, 2023
@arpad-m
Update azure SDK to 0.18 and use open range support (#6103)
7c2c87a 
* Update `azure-*` crates to 0.18
* Use new open ranges support added by upstream in
Azure/azure-sdk-for-rust#1482

Part of #5567. Prior update PR: #6081
This was referenced Apr 29, 2024
Open
Closed
Merged
arpad-m added a commit that referenced this issue May 3, 2024
@arpad-m
Increase Azure parallelism limit to 100 (#7597)
d769636 
After #5563 has been addressed we can now set the Azure strorage
parallelism limit to 100 like it is for S3.

Part of #5567
conradludgate pushed a commit that referenced this issue May 8, 2024
@arpad-m @conradludgate
Increase Azure parallelism limit to 100 (#7597)
c9b4be8 
After #5563 has been addressed we can now set the Azure strorage
parallelism limit to 100 like it is for S3.

Part of #5567
@arpad-m arpad-m mentioned this issue May 31, 2024
Open
5 tasks
@fcdm fcdm changed the title Epic: production ready Azure blob storage support Epic: [Azure] production ready Azure blob storage support Jun 21, 2024
@arpad-m arpad-m mentioned this issue Jul 17, 2024
Open
@arpad-m
Copy link
Member Author

arpad-m commented Jul 17, 2024

closing in favour of the follow-up in https://github.com/neondatabase/cloud/issues/12627 . The existing azure blob implementation is production-ready, the missing parts are in maintenance tooling outside of the main pageserver binary.

@arpad-m arpad-m closed this as completed Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arpad-m arpad-m

Labels
c/storage/pageserver Component: storage: pageserver c/storage/safekeeper Component: storage: safekeeper f/azure t/Epic Issue type: Epic team/azure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@stepashka @vadim2404 @seymourisdead @arpad-m