GH-37511: [C++] Implement file reads for Azure filesystem

[Tom-Newton]

Rationale for this change

We want a C++ implementation of an Azure filesystem. Reading files is the first step.

What changes are included in this PR?

Adds an implementation of io::RandomAccessFile for Azure blob storage (with or without hierarchical namespace (HNS) a.k.a datalake gen 2). This is largely copied from #12914. Using this io::RandomAccessFile implementation we implement the input file and stream methods of the AzureFileSystem.

I've made a few changes to the implementation from #12914. The biggest one is removing use of the Azure SDK datalake APIs. These APIs cannot be tested with azurite, they are only beneficial for listing operations on HNS enabled accounts and detecting a HNS enabled account is quite difficult (unless you use significantly elevated Azure permissions). Adding 2 different code paths for normal blob storage and datalake gen 2 seems like a bad idea to me except in cases where there is a performance advantage. I also made a few other tweaks to some of the error handling and to make things more consistent with the S3 or GCS filesystems.

Are these changes tested?

Yes. The tests are all based on the tests from the GCS filesystem with minimal chantges. I remember reading a review comment on #12914 which recommended this approach.
There are a few places where the GCS tests relied on file writes or file info methods so I've replaced those with direct calls to the Azure blob client and left TODO comments saying to switch them to use the AzureFilesystem when the relevant methods are implemented.

Are there any user-facing changes?

Yes. File reads using the Azure filesystem are now supported.

• Closes: [C++] Implement file reads for Azure filesystem #37511

[github-actions]

⚠️ GitHub issue #37511 has been automatically assigned in GitHub to PR creator.

[bkietz]

Thanks for working on this!

[bkietz]

Not necessary in the scope of this PR but FWIW this should be as simple as a call to BlobClient::GetProperties right?

[Tom-Newton]

Basically yes, but I was planning to do it in a separate PR to keep this as minimal as possible to file reads.

[Tom-Newton]

Actually it might require a check to determine if the storage account has hierarchical namespace enabled, at which point it could get quite complicated... but that is for another PR.

[Tom-Newton]

I had a look in a bit more detail. There will be a bit of logic required around the root of the container and the storage account but no need for anything too complicated for hierarchical namespace (ADLS gen2) storage accounts. I've create a github issue for it #38335

[felipecrv]

There is a formatting issue:

--- /arrow/cpp/src/arrow/filesystem/azurefs_test.cc
+++ /arrow/cpp/src/arrow/filesystem/azurefs_test.cc (after clang format)
@@ -308,8 +308,8 @@
 
   std::shared_ptr<const KeyValueMetadata> actual;
   ASSERT_OK_AND_ASSIGN(actual, stream->ReadMetadata());
-  // TODO(GH-38330): This is asserting that the user defined metadata is returned but this is 
-  // probably not the correct behaviour.
+  // TODO(GH-38330): This is asserting that the user defined metadata is returned but this
+  // is probably not the correct behaviour.
   ASSERT_OK_AND_EQ("value0", actual->Get("key0"));
 }
 
/arrow/cpp/src/arrow/filesystem/azurefs_test.cc had clang-format style issues

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 23dfd0e.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.