[Tracking Issue] Create/Convert ADLS Gen2 (data_lake) using Pipeline Architecture

[thovoll]

This is a tracking issue for creating/converting ADLS Gen2 (data_lake) operations using the pipeline architecture.

Overview

Previous to the pipeline architecture, each operation was independently controlled meaning there was no central way to control the way operations were performed.

The pipeline architecture on the other hand allows users of the SDK to create pipelines through which outgoing requests and incoming responses are processed. The pipeline are composed of "policies" which can both change the outgoing request and the incoming response. Policies are usually either on a "per-request" basis (meaning they are called only once per operation) or "per-retry" (meaning they are applied every time an operation is attempted).

Examples of policies are the TelemetryPolicy which adds various telemetry headers to the request and the retry policy which checks whether an incoming response was successful and if it detects a transient failure, retries the request.

Converting to the Architecture

For an example of adding the first operation using the pipeline architecture, take a look at #358 and its follow-ups #451 and #491.
Once the basic structure is established, adding new operations is easier: #495.

No existing operations have been converted yet, but the basic steps are shown in the CosmosDB tracking issue: #290.

Todos

• Files [datalake] Add DirectoryClient and FileClient #610

  • create_file add "create path" operation (files only for now) to Azure Storage data_lake (using Pipeline architecture) #358, data_lake/create_path improvements #451, Remove traits AsDataLakeClient and AsFileSystemClient #491
  • create_file_if_not_exists Add more data_lake operations #495
  • rename_file Add more data_lake operations #495
  • rename_file_if_not_exists Add rename_file_if_not_exists operation to data_lake #506
  • delete_file Add 'delete file' operation to data_lake. #512
  • lease_file
  • get_file_properties
  • get_file_status
  • get_file_access_control_list
  • read_file
  • append_to_file Add more data_lake operations #495
  • flush_file Add more data_lake operations #495
  • set_file_properties
  • set_file_access_control_list

• Directories [datalake] Add DirectoryClient and FileClient #610

  • create_directory
  • create_directory_if_not_exists
  • rename_directory
  • rename_directory_if_not_exists
  • delete_directory
  • get_directory_properties
  • get_directory_status
  • get_directory_access_control_list
  • lease_directory
  • list_directory
  • set_directory_properties
  • set_directory_access_control_list

• File Systems [datalake] Migrate file system operations to pipeline architecture #597

  • list_file_systems
  • create_file_system
  • delete_filesystem
  • get_file_system_properties
  • set_file_system_properties

Related Issues

• Add test using mock test framework for storage/data_lake #372
• Take TokenCredential instead of bearer token #426
• Remove azure_identity::TokenCredential trait in favor of azure_core::TokenCredential #489
• Allow creation of nested clients directly? #490
• Remove traits AsDataLakeClient and AsFileSystemClient #491

[roeap]

I just opened a PR for all the file system operations (#597), maybe that should be referenced in this Issue?

[roeap]

@thovoll - with #597 and #610 merged, we should be able to check some of these boxes ;)

• Files
  • read_file
  • get_file_properties
  • set_file_properties
• FileSystem
  • all
• Direcrtory
  • create_directory
  • create_directory_if_not_exists
  • rename_directory
  • rename_directory_if_not_exists
  • delete_directory
  • get_directory_properties

[cataggar]

It looks like #853 from @roeap just about completed this. The todos just mention a lease fn for the FileClient and DirectoryClient. Is that still needed to complete this? @thovoll

[roeap]

Not sure hot @thovoll views this, but I had a look at other SDKs, as well as blob over here, and lease seems to be consistently implemented using a dedicated LeaseClient. So I think file and directory client could be considered complete in terms of exposed functionality.

[roeap]

@cataggar - AFAIK #372 and #426 are done as well.

[cataggar]

If a LeaseClient is needed, please add a separate issue.