[C++] Filesystem implementation for Azure Blob Storage

[asfimport]

Subissues:

• [C++] Skeleton for Azure Filesystem Implementation #35903
• [C++] Configure azurite in preparation for testing Azure C++ filesystem #36886
• [C++] Implement file reads for Azure filesystem #37511
• [C++] Return filesystem properties not user defined metadata in Azure file reads #38330
• [C++] Implement file writes for Azure filesystem #38333
• [C++] Implement GetFileInfo for a single file in Azure filesystem #38335
• [C++] Implement GetFileInfo selector for Azure filesystem #38597
• [C++] Implement all important authentication methods for Azure filesystem #38598
  • [C++][FS][Azure] Default credential authentication #39262
  • [C++][FS][Azure] Workload Identity Authentication #39318
  • [C++][FS][Azure] Managed Identity authentication #39320
  • [C++][FS][Azure] Client secret authentication #39343
  • [C++][FS][Azure] Azure CLI authentation #39344
  • [C++][FS][Azure] Environment credential authentication #39345
  • [C++][FS] Use default Azure credentials implicitly and support anonymous credentials explictly #39449
• [C++][FS][Azure] Implement CreateDir() #38699
• [C++][FS][Azure] Implement DeleteDir() #38700
• [C++][FS][Azure] Implement DeleteDirContents() #38701
• [C++][FS][Azure] Implement DeleteRootDirContents() #38702
• [C++][FS][Azure] Implement DeleteFile() #38703
• [C++][FS][Azure] Implement DeleteFile() for flat-namespace storage accounts #40074
• [C++][FS][Azure] Implement Move() #38704
• [C++][FS][Azure] Implement CopyFile() #38705
• [C++][FS][Azure] Rename AzurePath to AzureLocation #38758
• [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772
• [C++][FS][Azure] Add allow_container_deletion option #38999
• [C++][FS][Azure] Use the generic filesystem tests #39069
  • [C++][FS][Azure] Generic filesystem tests for DeleteDir/DeleteDirContents/GetFileInfoSelector failed with Azure #41034
• [C++][FS][Azure] Make mutations of test data explicit in azurefs_test.cc #39119
• [C++] Remove the AzureOptions::backend option #39292
• [C++] Inform caller of Azure Storage container not-existing when checking for HNS support #39297
• [C++][FS][Azure] Implement Move() for flat namespace storage accounts #40025
• [C++][FS][Azure] Include user set metadata in ObjectInputFile::ReadMetadata #40026
• [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028
• [C++][FS][Azure] Expose parallel transfer config options available in the Azure SDK #40035
• [C++][FS][Azure] Optimise ObjectAppendStream::DoAppend in the case of many small appends #40036
• [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037
• [C++][FS][Azure] Fix CreateDir and DeleteDir on hierarchical namespace accounts where the directory has a trailing / #40052
• [C++][FS][Azure] Consider how and if to fix disallowed characters in file metadata #40057
• ...

---

Reporter: Wes McKinney / @wesm
Assignee: Shefali Singh

Related issues:

• [C++] Filesystem implementation for Azure Data Lake (is cloned by)
• [R] Expose Azure Blob Storage filesystem (is depended upon by)

Original Issue Attachments:

• azfs.cc
• azfs.h

PRs and other links:

• GitHub Pull Request #12914

_{Note: This issue was originally created as ARROW-2034. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
I see that TileDB (MIT license) has built a C++ wrapper for Azure

https://github.com/TileDB-Inc/TileDB/blob/dev/tiledb/sm/filesystem/azure.cc

No this is not moved to fsspec, this is a C++ ticket

[asfimport]

Antoine Pitrou / @pitrou:
I'm not sure I understand the relationship between Blob Store and Data Lake. Is Data Lake a higher-level layer above Blob Store? Or are they two different services that would need separate filesystem implementations?

[asfimport]

Antoine Pitrou / @pitrou:
According to https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction ,

Data Lake Storage Gen2 converges the capabilities of Azure Data Lake Storage Gen1 with Azure Blob storage. For example, Data Lake Storage Gen2 provides file system semantics, file-level security, and scale. Because these capabilities are built on Blob storage, you'll also get low-cost, tiered storage, with high availability/disaster recovery capabilities.
I'm not sure this means the same C++ API can be used to access both, though.

[asfimport]

Uwe Korn / @xhochy:
It is as confusing in reality, here is what they all are (I'm though already 1 year outdated on this):

• Blob Store: Like S3, simple but limited API
• Data Lake Gen 1: HDFS-like deployment with different but more user-friendly API / attributes
• Data Lake Gen 2: Some improvements were made to the Blob Store so that there is no need for a special (more expensive) Data Lake service anymore, everything is now on the Blob Store. A new set of APIs was though released that exposes some nice features that the initial Blob Store API didn't have, probably for marketing purposes this was named Data Lake Gen 2 although technically Blob Store Gen 2 would have been more appropriate.

[asfimport]

Antoine Pitrou / @pitrou:
Ha, and here are some interesting resources:

• https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues : "Blob APIs and Data Lake Storage Gen2 APIs can operate on the same data." but "You cannot use blob API and Data Lake Storage APIs to write to the same instance of a file" (sounds ok?)

• https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs : three different kinds of blobs. "Block blobs" sound most useful, though "append blobs" may work too.

• https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace : a Data Lake admin can configure a it with a hierarchical filesystem semantics.  Apparently it may enable different APIs (??).

   

   

[asfimport]

Yesh:
There is also https://github.com/Azure/azure-sdk-for-cpp which I’ve tested against adl gen2 .

[asfimport]

Antoine Pitrou / @pitrou:
Various pointers about the Azure C++ SDK:

https://devblogs.microsoft.com/azure-sdk/cppintro/#example-code-using-the-c-storage-blob-client-libraryhttps://devblogs.microsoft.com/azure-sdk/cppintro/

https://github.com/Azure/azure-sdk-for-cpp

https://github.com/Azure/azure-sdk-for-cpp/blob/master/sdk/storage/azure-storage-blobs/sample/blob_getting_started.cpp

https://azure.github.io/azure-sdk-for-cpp/ (note the API docs are segregated per component, with separate land pages for Core, Storage, Blog Storage, etc.)

Note that the SDK requires C++14.

 

[asfimport]

Tom Augspurger / @TomAugspurger:
Does Arrow support C+\14 features now (or more specifically, is the SDK being C\14 a problem?) From https://issues.apache.org/jira/browse/ARROW-13744 it seems like C\14 is at least tested, but https://github.com/apache/arrow/blame/master/docs/source/developers/cpp/building.rst#L40 says a "A C\+11-enabled compiler. " is required.

[asfimport]

Neal Richardson / @nealrichardson:
I'm not an expert here, but I think we could use C+14 if required and if the compiler supports it. If the compiler doesn't support C 14, we wouldn't be able to build the azure sdk. So the line would be that Arrow requires C 11 at a minimum, and some features are only available with C+14.

[asfimport]

Antoine Pitrou / @pitrou:
We require only C+11 in the codebase. We might add C+14-requiring optional components if desired, but that will add complication to the build setup.

[asfimport]

Shashanka Balakuntala Srinivasa:
hi @pitrou , we were looking into implementing this feature from our side. I did try to compile the whole arrow code base with c+14 and ran unit tests as well. Everything is passing in local and as mentioned before : [ARROW-13744] [CI] c+14 and 17 nightly job fails - ASF JIRA (apache.org) ticket mentions we have daily build run for validation which are passing. 

Since the azure sdks depend on c+14 features, and since we have the code compiled in c 14, can we look into upgrading the c+ version to 14? 
Let me know if there are any issues. I will be happy to take those and work on them.

[asfimport]

Antoine Pitrou / @pitrou:
[~balakuntala] We can probably require C+\14 for the Azure filesystem implementation only, but the rest of Arrow should remain C+11-compatible.

[asfimport]

Dean MacGregor:
If someone wants to work on this but doesn't have an Azure account let me know.  I can make a storage account for this development/testing

[asfimport]

Bipin Mathew:
I have a rudimentary implementation of this that supports import and export to Azure. I tried to align as closely as possible to the s3 implementation, however as I needed it only for a specific use case and have not had a chance to implement all the same methods. Depending on bandwidth, I could probably build out the implementation further. I have never contributed to this project. Do all endpoints need to be implemented before it can be merged into the code base? Can it be built out over time? I attach what I have so far here.

azfs.hazfs.cc

[asfimport]

Antoine Pitrou / @pitrou:
[~mathewb001] Well, there's a Github PR open already, did you take a look?

[asfimport]

Bipin Mathew:
Oh this is much further along than what I have offered. Looking forward to
its release so I can cut over to it.

[av8or1]

We too are in need of this feature. Any word as to whether it will be included in the version 11 release? Seems like the deadline is kinda tight at this point.

[pitrou]

@h-vetinari Do you know if there's a conda-forge package for the Azure Blob Storage C++ library? I couldn't find one.

[h-vetinari]

@h-vetinari Do you know if there's a conda-forge package for the Azure Blob Storage C++ library? I couldn't find one.

I'm not aware of any either, but I don't know. What would be the sources of that C++ lib? If it's open source we could bring it to conda-forge eventually of course.

[pitrou]

@h-vetinari I think it's this: https://github.com/Azure/azure-sdk-for-cpp/tree/main

[raulcd]

I am moving this umbrella issue to 16.0.0

[felipecrv]

I am moving this umbrella issue to 16.0.0

That's fine. Thanks.

[av8or1]

kou/felipecrv - How close is this to being done? I see a few green-colored items in the list above, but they seem to be completed already (at least the ones I looked at did). Is there anything else we can work on? Will this make it into version 16.0.0? Thanks

[felipecrv]

@av8or1 most of it works now. s3fs doesn't even support Move. AzureFileSystem supports Move on accounts with HNS enabled. You don't have to wait for this issue to be closed to start using what's already merged and ready to be part of the 16.0.0 release.

[av8or1]

Hi felipe- Thanks. Well, the company can't utilize any library that isn't an official release. I suppose that I could begin writing the code that will utilize the ADLS stuff now, then when 16.0.0 is released (April?), I would be able to produce our product shortly thereafter. What remains to be completed, by the way? Anything I could help with? Thanks

[felipecrv]

@av8or1 as I said above: Move with Blobs API (not a critical feature at all, s3fs doesn't even support Move). Python bindings (PR is open), and URI parsing (PR is open). Are any of these a dealbreaker for you? Everything else will be available in 16.0.0.

[av8or1]

@felipecrv OK thank you. Work has been busy. Just now looking at this again. It appears that @kou has completed the URI parsing business (#40028). Thus I will prepare on my end to use the library when it is released. Hopefully in April. Thanks

[wirable23]

Everything else will be available in 16.0.0.

@felipecrv do you know when 16.0.0 would be available?

[felipecrv]

Everything else will be available in 16.0.0.

@felipecrv do you know when 16.0.0 would be available?

In April. The time it takes for the release also depends on how smooth the packaging and publishing process goes.

[Tom-Newton]

I think ideally #40036 would be taken care of before the 16.0.0 release. I don't have any real world performance numbers but I suspect write performance is currently a bit disappointing.

[raulcd]

Hi @Tom-Newton there's no planned release before 16.0.0
The feature freeze for 16.0.0 is planned for the 8th of April.

[felipecrv]

I opened a MINOR PR expanding some of the docstrings: #40838

[raulcd]

There are some subtasks still opened. @felipecrv should I tag this umbrella issue as 17.0.0?

[felipecrv]

There are some subtasks still opened. @felipecrv should I tag this umbrella issue as 17.0.0?

AzureFileSystem is already usable and feature-complete even without these open issues being fixed. I'm in favor of closing (marking it as complete) making it part of the 16.0 release in the logs.

[kou]

Let's close this as complete.
We don't need an umbrella issue for AzureFileSystem. We can just use separated issues like other components.

[kou]

Note that the current AzureFileSystem's CopyFile() doesn't work with Azure hierarchical namespace support. See also: #41095

Some other AzureFileSystem implementations for Azure hierarchical namespace support have some problems: #41034
I want to add the fix of this (#41068) to 16.0.0.