GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40325

kou · 2024-03-03T09:34:40Z

Rationale for this change

FileSystemFromUri() is a common API to create a file system object. FileSystemFromUri() should be able to create an AzureFileSystem object.

What changes are included in this PR?

Add AzureOptions::FromUri() and use it from FileSystemFromUri().

See the AzureOptions::FromUri()'s docstring about the supported formats.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

GitHub Issue: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

…ystemFromUri()

github-actions · 2024-03-03T09:35:07Z

⚠️ GitHub issue #40028 has been automatically assigned in GitHub to PR creator.

kou · 2024-03-03T09:35:36Z

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

Tom-Newton · 2024-03-03T09:53:27Z

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

nosterlu · 2024-03-03T11:34:56Z

@Tom-Newton Are abfs:// and abfss:// natural for AzureFileSystem?

I would say yes. I think originally they came from Hadoop with the extra "s" indicating secure but other filesystem implementations seem to have adopted it and they seem to be used mostly interchangeably.

From the documentation if it helps

The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

kou · 2024-03-03T20:36:18Z

Thanks for the note. I've added the documentation URL as a comment.

felipecrv · 2024-03-04T17:38:48Z

cpp/src/arrow/filesystem/azurefs.cc

+Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
+                                           std::string* out_path) {


A PR by @bkietz is moving Uri out of internal so we should be careful with the merges.

Thanks for the info!
#39067

felipecrv · 2024-03-04T17:44:36Z

cpp/src/arrow/filesystem/azurefs.cc

+  if (container.empty()) {
+    return Status::Invalid("Missing container name in Azure Blob File System URI");
+  }


Why you need a container name if the filesystem wraps the entire storage account?

Ah, we don't need this check. I'll remove this.
(I used GcsOptions::FromUri() as a base implementation and forgot to remove this check.)

felipecrv · 2024-03-04T17:48:19Z

cpp/src/arrow/filesystem/azurefs.cc

+  std::unordered_map<std::string, std::string> options_map;
+  ARROW_ASSIGN_OR_RAISE(const auto options_items, uri.query_items());
+  for (const auto& kv : options_items) {
+    options_map.emplace(kv.first, kv.second);
+  }


No need to build the map if you're going to iterate over the kv pairs and switch. This is just randomizing the iteration order.

You're right. I borrowed this implementation from GcsOptions::FromUri() but I should have noticed this.
(We should remove this conversion in GcsOptions::FromUri() too later.)

felipecrv · 2024-03-04T17:50:39Z

cpp/src/arrow/filesystem/azurefs.cc

+      options.blob_storage_scheme = kv.second;
+    } else if (kv.first == "dfs_storage_scheme") {
+      options.dfs_storage_scheme = kv.second;
+    } else if (kv.first == "credential_kind") {


credential_kind_ should be inferred from what you find on the URI without the user having to set both the credential kind and the credentials.

Does it mean that we should use ConfigureClientSecretCredential() if tenant_id, client_id and client_secret are specified but credential_kind=client_secret isn't specified?

credential_kind should never be specified and we should validate the URI to keep the invariant that it doesn't configure two different auth methods. And when nothing is provided, we use the default auth chain provided by the SDK.

Hmm, how can we distinguish ConfigureAnonymousCredential(), ConfigureWorkloadIdentityCredential() and ConfigureDefaultCredential()? All of them don't require additional information.

The parameter-less auth methods can have dedicated query params for each. These being the valid configurations regarding auth:

nothing (use default auth chain)

?anonymous

?use_workload_identity

?account_key=<ACCOUNT_KEY>

?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET>

?client_id=<CLIENT_ID> (client_id alone means managed identity credential)

?anonymous and ?use_workaround_identity are conflicted parameters. (We can't specify both of them at once.) I think that it's better that we use the same parameter name for the type (XXX={anonymous,workload_identity}). If we use it, users can't specify both of them at once. (I know that URI spec accepts XXX=anonymous&XXX=workload_identity.)

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

nothing (use default auth chain) -> nothing or ?credential_kind=default

?anonymous -> ?credential_kind=anonymous

?use_workload_identity -> ?credential_kind=workload_identity

?account_key=<ACCOUNT_KEY> -> not changed (?credential_kind=storage_shared_key is invalid)

?tenant_id=<TENANT_ID>&client_id=<CLIENT_ID>&client_secret=<CLIENT_SECRET> -> not changed (?credential_kind=client_secret is invalid)

?client_id=<CLIENT_ID> (client_id alone means managed identity credential) -> not changed (?credential_kind=managed_identity is invalid)

Ah, we don't need ?account_key=<ACCOUNT_KEY> because we can get it from the URI's password part.

How about accepting only (default, ) anonymous and use_workload_identity as valid credential_kind parameter?

Sure. That looks good.

cpp/src/arrow/filesystem/azurefs.h

cpp/src/arrow/filesystem/azurefs.cc

cpp/src/arrow/filesystem/filesystem.cc

felipecrv · 2024-03-04T18:25:25Z

@kou @Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

Tom-Newton · 2024-03-04T18:53:03Z

@kou @Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

felipecrv · 2024-03-04T19:40:49Z

@kou @Tom-Newton can we use azfs (only https and by default) because this implementation uses both Blobs and File System APIs?

I have quite a strong opinion that we should support the abfss:// and abfs:// Hadoop path formats. As a user I don't want to use different path formats when I use different filesystem implementations.

Fair enough, then what are the semantics of each URI format and how they map to this implementation?

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions cause people running pyarrow on insecure Wi-Fi networks shouldn't have their credentials leaked.

Tom-Newton · 2024-03-04T20:09:39Z

I will start with one rule: both abfss and abfs map to scheme=https in AzureOptions

That is fine with me.

what are the semantics of each URI format and how they map to this implementation?

For Hadoop format URIs I would probably just extract the storage account name. That means there is a lot of redundant information in the URI but I don't think that is really a problem and it gives us compatibility which I think is important.

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url https://github.com/fsspec/adlfs

felipecrv · 2024-03-05T15:25:06Z

If we want to support other URIs formats too that would be useful to some people. Some examples I've seen on other filesystems: https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url https://github.com/fsspec/adlfs

Thank you @Tom-Newton! This list from the Rust crate defines formats that allow us to express URIs that refer to the entire storage account and not just a specific filesystem. Plus, simple and short URIs that allow us to simply use the default endpoints. cc @kou

kou · 2024-03-06T08:07:39Z

URI list from https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_url :

abfs[s]://<container>/<path> (according to fsspec)
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
az://<container>/<path>
adl://<container>/<path>
azure://<container>/<path>
https://<account>.dfs.core.windows.net
https://<account>.blob.core.windows.net
https://<account>.blob.core.windows.net/<container>
https://<account>.dfs.fabric.microsoft.com
https://<account>.dfs.fabric.microsoft.com/<container>
https://<account>.blob.fabric.microsoft.com
https://<account>.blob.fabric.microsoft.com/<container>

We can use abfs/abfss/az/adl/azure schemes with the current FileSystemFromUri() mechanism. But https is a bit tricky. We need to check not only the scheme part but also the host part. This means that we need to put Azure related URI parsing code to filesystem.cc and azurefs.cc. And this isn't suitable for #39067 . #39067 only uses the scheme part to dispatch filesystem implementation.

felipecrv · 2024-03-06T13:52:06Z

@kou let's implement only abfs[s] for now. Not even use the az, adl, azure schemes.

abfs[s]://<container>/<path> (according to [fsspec](https://github.com/fsspec/adlfs))
abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>

Supporting https:// would onlye make sense if we were creating an Object Store API, but we are implementing filesystem abstractions on top of object stores accessible via HTTP, so it makes more sense to have them named something that is not http[s].

kou · 2024-03-06T21:55:28Z

Can we also support one more format for Azurite?

abfs[s]://<host>:<port>/<container>/<path>

If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

felipecrv · 2024-03-06T22:03:33Z

Can we also support one more format for Azurite?
abfs[s]://<host>:<port>/<container>/<path>
If <host> doesn't have . and the <port> part doesn't exist, we will interpret the given URI as abfs[s]://<container>/<path>.

Sure. I think that can work well.

kou · 2024-03-07T00:00:25Z

OK. I'll implement the discussed spec.

Supported formats: 1. abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]] 2. abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path] 3. abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]] 4. abfs[s]://[<account[:<password>]@]<container>[/path] Added query parameters: * enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme parameters. Removed query parameters: * blob_storage_scheme: Replaced with enable_tls. * dfs_storage_scheme: Replaced with enable_tls. Changed query parameters: * credential_kind: Accepts only "default", "anonymous" and "workload_identity".

kou · 2024-03-07T03:01:40Z

Implemented:

Supported formats:

abfs[s]://[:<password>@]<account>.blob.core.windows.net[/<container>[/<path>]]
abfs[s]://<container>[:<password>]@<account>.dfs.core.windows.net[/path]
abfs[s]://[<account[:<password>]@]<host[.domain]>[<:port>][/<container>[/path]]
abfs[s]://[<account[:<password>]@]<container>[/path]

Added query parameters:

enable_tls: It replaces blob_storage_scheme and dfs_storage_scheme
parameters.

Removed query parameters:

blob_storage_scheme: Replaced with enable_tls.
dfs_storage_scheme: Replaced with enable_tls.

Changed query parameters:

credential_kind: Accepts only default, anonymous and
workload_identity.

kou · 2024-03-08T08:08:03Z

I'll merge this in the next week if nobody objects it.

conbench-apache-arrow · 2024-03-12T17:46:15Z

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 605f8a7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change Failure to rebase and build when merging #39067 (which renamed `internal::Uri` -> `util::Uri`) led to a merge conflict since #40325 added more usages of `internal::Uri` ### What changes are included in this PR? Rename internal::Uri -> util::Uri ### Are these changes tested? Yes ### Are there any user-facing changes? No * GitHub Issue: #40562 Authored-by: Benjamin Kietzman <[email protected]> Signed-off-by: Benjamin Kietzman <[email protected]>

apacheGH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileS…

704b69e

…ystemFromUri()

kou requested a review from felipecrv March 3, 2024 09:34

github-actions bot added Component: C++ awaiting committer review Awaiting committer review labels Mar 3, 2024

kou added 2 commits March 4, 2024 05:29

Update example URIs

9e7d75e

Add the document URI for abfs:// and abfss://

81076bc

felipecrv requested changes Mar 4, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Mar 5, 2024

Remove needless empty container check

fb692de

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 5, 2024

Remove needless options_map

376017f

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 6, 2024

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 7, 2024

kou added 2 commits March 7, 2024 22:05

Escape special characters

5927be8

Fix lint error

0a76645

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 7, 2024

Fix lint error

58d948f

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 8, 2024

kou merged commit 605f8a7 into apache:main Mar 11, 2024
34 of 35 checks passed

kou deleted the cpp-azurefs-from-uri branch March 11, 2024 21:34

kou removed the awaiting change review Awaiting change review label Mar 11, 2024

kou mentioned this pull request Mar 11, 2024

[C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028

Closed

This was referenced Mar 14, 2024

[C++] Merge conflict with AzureFS changes #40562

Closed

GH-40562: [C++] Repair FileSystem merge error #40564

Merged

Tom-Newton mentioned this pull request Mar 15, 2024

[Python][FS][Azure] Expose all the configuration options to python #40572

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40325

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40325

kou commented Mar 3, 2024 •

edited

Loading

github-actions bot commented Mar 3, 2024

kou commented Mar 3, 2024

Tom-Newton commented Mar 3, 2024

nosterlu commented Mar 3, 2024 •

edited

Loading

kou commented Mar 3, 2024

felipecrv Mar 4, 2024

kou Mar 5, 2024

felipecrv Mar 4, 2024

kou Mar 5, 2024

felipecrv Mar 4, 2024

kou Mar 5, 2024

felipecrv Mar 4, 2024

kou Mar 5, 2024

felipecrv Mar 5, 2024

kou Mar 6, 2024

felipecrv Mar 6, 2024

kou Mar 7, 2024

kou Mar 7, 2024

felipecrv Mar 7, 2024

felipecrv commented Mar 4, 2024

Tom-Newton commented Mar 4, 2024 •

edited

Loading

felipecrv commented Mar 4, 2024 •

edited

Loading

Tom-Newton commented Mar 4, 2024 •

edited

Loading

felipecrv commented Mar 5, 2024

kou commented Mar 6, 2024

felipecrv commented Mar 6, 2024 •

edited

Loading

kou commented Mar 6, 2024

felipecrv commented Mar 6, 2024

kou commented Mar 7, 2024

kou commented Mar 7, 2024

kou commented Mar 8, 2024

conbench-apache-arrow bot commented Mar 12, 2024

		Result<AzureOptions> AzureOptions::FromUri(const arrow::internal::Uri& uri,
		std::string* out_path) {

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40325

GH-40028: [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40325

Conversation

kou commented Mar 3, 2024 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Mar 3, 2024

kou commented Mar 3, 2024

Tom-Newton commented Mar 3, 2024

nosterlu commented Mar 3, 2024 • edited Loading

kou commented Mar 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felipecrv commented Mar 4, 2024

Tom-Newton commented Mar 4, 2024 • edited Loading

felipecrv commented Mar 4, 2024 • edited Loading

Tom-Newton commented Mar 4, 2024 • edited Loading

felipecrv commented Mar 5, 2024

kou commented Mar 6, 2024

felipecrv commented Mar 6, 2024 • edited Loading

kou commented Mar 6, 2024

felipecrv commented Mar 6, 2024

kou commented Mar 7, 2024

kou commented Mar 7, 2024

kou commented Mar 8, 2024

conbench-apache-arrow bot commented Mar 12, 2024

kou commented Mar 3, 2024 •

edited

Loading

nosterlu commented Mar 3, 2024 •

edited

Loading

Tom-Newton commented Mar 4, 2024 •

edited

Loading

felipecrv commented Mar 4, 2024 •

edited

Loading

Tom-Newton commented Mar 4, 2024 •

edited

Loading

felipecrv commented Mar 6, 2024 •

edited

Loading