400 Error when using FileClient.Delete() with ApiVersion[2023-08-03] and OAuth in Storage Datalake SDK

[microzchang]

Library name and version

Azure.Storage.Files.Datalake.12.16.0.alpha.20230726.1

Describe the bug

In the latest source code of Storage Datalake SDK, when using FileClient.Delete() with ApiVersion2023-08-03 and OAuth, the query parameter of the request should be paginated=true&recursive=false. This combination produces a 400 error no matter if the resourceType is file or directory.

Expected behavior

Return success instead of 400 error.

Actual behavior

Return HttpStatusCode 400 with message: A query parameter that's mandatory for this request is not specified.

Reproduction Steps

1. Use the latest source code with include feature Paginated Directory Delete to build.
2. Use FileClient.Delete() with ApiVersion2023-08-03 and OAuth

There is also a x-ms-reuqest-id about this issue to search server log: c2e472e5-101f-0047-3430-c026fb000000

Environment

Visual Studio 2022

[github-actions]

Thank you for your feedback. This has been routed to the support team for assistance.

[seanmcc-msft]

The paginated and recursive query parameters do not apply to file delete - https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete

[microzchang]

Hi, Sean.
Thanks for your reply,I think The paginated and recursive query parameters do not apply to file delete is right.
But in the impl of .Net SDK, DatalakeFileClient.Delete() will use DtalakePathClient.Delete(), which will set paginated=true when ApiVersion = 2023-08-03 and TokenCredential is not null. This will trigger a 400 error.

You can easily reproduce this 400 error by using following code:

DataLakeClientOptions options = new DataLakeClientOptions(DataLakeClientOptions.ServiceVersion.V2023_08_03);

//AAD with RBAC permission
ClientSecretCredential credential = new ClientSecretCredential("AAD_TENANT_ID", "AAD_CLIENT_ID", "AAD_SCERET");


var oauthService = new DataLakeServiceClient(new Uri("https://zchangdatalakecanary.dfs.core.windows.net"), credential, options);
var oauthFileSystem = oauthService.GetFileSystemClient("test1");
oauthFileSystem.CreateIfNotExists();
var oauthFile = oauthFileSystem.GetFileClient("file1");
oauthFile.CreateIfNotExists();
oauthFile.Delete();

oauthFileSystem.Delete();

And you can get the following error:

 Message: 
Azure.RequestFailedException : A query parameter that's mandatory for this request is not specified.
RequestId:da28406e-001f-0039-7bf9-c0b6bc000000
Time:2023-07-28T02:14:35.8452650Z
Status: 400 (A query parameter that's mandatory for this request is not specified.)
ErrorCode: MissingRequiredQueryParameter

Content:
{"error":{"code":"MissingRequiredQueryParameter","message":"A query parameter that's mandatory for this request is not specified.\nRequestId:da28406e-001f-0039-7bf9-c0b6bc000000\nTime:2023-07-28T02:14:35.8452650Z"}}

Headers:
Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0
x-ms-error-code: MissingRequiredQueryParameter
x-ms-request-id: da28406e-001f-0039-7bf9-c0b6bc000000
x-ms-version: 2023-08-03
x-ms-client-request-id: 03e333ce-5310-4fed-84a2-d2859361ee2d
Date: Fri, 28 Jul 2023 02:14:34 GMT
Content-Length: 215
Content-Type: application/json;charset=utf-8


  Stack Trace: 
PathRestClient.Delete(Nullable`1 timeout, Nullable`1 recursive, String continuation, String leaseId, String ifMatch, String ifNoneMatch, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, Nullable`1 paginated, CancellationToken cancellationToken) line 946
DataLakePathClient.DeleteInternal(Nullable`1 recursive, DataLakeRequestConditions conditions, Boolean async, CancellationToken cancellationToken) line 1891
TaskExtensions.EnsureCompleted[T](Task`1 task) line 33
DataLakePathClient.Delete(Nullable`1 recursive, DataLakeRequestConditions conditions, CancellationToken cancellationToken) line 1763
DataLakeFileClient.Delete(DataLakeRequestConditions conditions, CancellationToken cancellationToken) line 931
Sample01a_HelloWorld.CreateFileClient_Filesystem() line 48

[microzchang]

And another thing is that the TokenCredential may not pass correctly to _clientConfiguration when directly using following API to create FileClient/DirectoryClient:

ClientSecretCredential credential = new ClientSecretCredential("AAD_TENANT_ID", "AAD_CLIENT_ID", "AAD_SCERET");

var oauthFile = new DataLakeFileClient(new Uri("DFS_FILE_URI"), credential, options);
var oauthDirectory = new DataLakeDirectoryClient(new Uri("DFS_DIRECTORY_URI"), credential, options);

The TokenCredential field in _clientConfiguration of FileClient/DirectoryClient will be empty.

The code line:
FileClient
DirectoryClient

This can cause the value of paginated to be set incorrectly when Delete().

[navba-MSFT]

Adding Service team to look into this

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sumantmehtams.

[seanmcc-msft]

This was fixed with #37934

[seanmcc-msft]

The 2nd issue was fixed with #38083