From 1ddb616fe1cc7cfa55b3cecd93e25f68f15730cb Mon Sep 17 00:00:00 2001 From: Cole MacKenzie Date: Wed, 6 Mar 2024 19:49:15 -0800 Subject: [PATCH] docs: update comment about r2 requiring locks Cloudflare R2 doesn't require the use of an external lock when using conditional operation headers like so: ```rust let s3 = AmazonS3Builder::from_env() .with_url(url) .with_region("auto") .with_access_key_id(&config.s3_access_key_id) .with_secret_access_key(&config.s3_secret_access_key) .with_endpoint(&config.s3_endpoint) // Allows using S3-API without an external locking provider since Cloudflare R2 // provides atomic Put and Copy. .with_config( AmazonS3ConfigKey::CopyIfNotExists, "header: cf-copy-destination-if-none-match: *".to_string(), ) .build() ``` - https://developers.cloudflare.com/r2/api/s3/extensions/#putobject-1 - https://github.com/apache/arrow-rs/blob/c6ba0f764a9142b74c9070db269de04d2701d112/object_store/src/aws/precondition.rs#L29-L42 --- README.md | 64 +++++++++++++++++++++++++++---------------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 2c10705be4..100d34355d 100644 --- a/README.md +++ b/README.md @@ -130,45 +130,45 @@ of features outlined in the Delta [protocol][protocol] is also [tracked](#protoc ### Cloud Integrations -| Storage | Rust | Python | Comment | -| -------------------- | :-----: | :-----: | ----------------------------------- | -| Local | ![done] | ![done] | | -| S3 - AWS | ![done] | ![done] | requires lock for concurrent writes | -| S3 - MinIO | ![done] | ![done] | requires lock for concurrent writes | -| S3 - R2 | ![done] | ![done] | requires lock for concurrent writes | -| Azure Blob | ![done] | ![done] | | -| Azure ADLS Gen2 | ![done] | ![done] | | -| Microsoft OneLake | ![done] | ![done] | | -| Google Cloud Storage | ![done] | ![done] | | +| Storage | Rust | Python | Comment | +| -------------------- | :-----: | :-----: | ---------------------------------------------------------------- | +| Local | ![done] | ![done] | | +| S3 - AWS | ![done] | ![done] | requires lock for concurrent writes | +| S3 - MinIO | ![done] | ![done] | requires lock for concurrent writes | +| S3 - R2 | ![done] | ![done] | No lock required when using `AmazonS3ConfigKey::CopyIfNotExists` | +| Azure Blob | ![done] | ![done] | | +| Azure ADLS Gen2 | ![done] | ![done] | | +| Microsoft OneLake | ![done] | ![done] | | +| Google Cloud Storage | ![done] | ![done] | | ### Supported Operations -| Operation | Rust | Python | Description | -| --------------------- | :----------------------: | :----------------------: | ------------------------------------------- | -| Create | ![done] | ![done] | Create a new table | -| Read | ![done] | ![done] | Read data from a table | -| Vacuum | ![done] | ![done] | Remove unused files and log entries | -| Delete - partitions | | ![done] | Delete a table partition | -| Delete - predicates | ![done] | ![done] | Delete data based on a predicate | -| Optimize - compaction | ![done] | ![done] | Harmonize the size of data file | -| Optimize - Z-order | ![done] | ![done] | Place similar data into the same file | -| Merge | ![done] | ![done] | Merge a target Delta table with source data | -| FS check | ![done] | ![done] | Remove corrupted files from table | +| Operation | Rust | Python | Description | +| --------------------- | :-----: | :-----: | ------------------------------------------- | +| Create | ![done] | ![done] | Create a new table | +| Read | ![done] | ![done] | Read data from a table | +| Vacuum | ![done] | ![done] | Remove unused files and log entries | +| Delete - partitions | | ![done] | Delete a table partition | +| Delete - predicates | ![done] | ![done] | Delete data based on a predicate | +| Optimize - compaction | ![done] | ![done] | Harmonize the size of data file | +| Optimize - Z-order | ![done] | ![done] | Place similar data into the same file | +| Merge | ![done] | ![done] | Merge a target Delta table with source data | +| FS check | ![done] | ![done] | Remove corrupted files from table | ### Protocol Support Level -| Writer Version | Requirement | Status | -| -------------- | --------------------------------------------- | :------------------: | -| Version 2 | Append Only Tables | ![done] | -| Version 2 | Column Invariants | ![done] | -| Version 3 | Enforce `delta.checkpoint.writeStatsAsJson` | [![open]][writer-rs] | -| Version 3 | Enforce `delta.checkpoint.writeStatsAsStruct` | [![open]][writer-rs] | +| Writer Version | Requirement | Status | +| -------------- | --------------------------------------------- | :-------------------------------: | +| Version 2 | Append Only Tables | ![done] | +| Version 2 | Column Invariants | ![done] | +| Version 3 | Enforce `delta.checkpoint.writeStatsAsJson` | [![open]][writer-rs] | +| Version 3 | Enforce `delta.checkpoint.writeStatsAsStruct` | [![open]][writer-rs] | | Version 3 | CHECK constraints | [![semi-done]][check-constraints] | -| Version 4 | Change Data Feed | | -| Version 4 | Generated Columns | | -| Version 5 | Column Mapping | | -| Version 6 | Identity Columns | | -| Version 7 | Table Features | | +| Version 4 | Change Data Feed | | +| Version 4 | Generated Columns | | +| Version 5 | Column Mapping | | +| Version 6 | Identity Columns | | +| Version 7 | Table Features | | | Reader Version | Requirement | Status | | -------------- | ----------------------------------- | ------ |