Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++][FS][Azure] CopyFile doesn't work with Azure hierarchical namespace support #41095

Closed
kou opened this issue Apr 9, 2024 · 7 comments
Closed

[C++][FS][Azure] CopyFile doesn't work with Azure hierarchical namespace support #41095

kou opened this issue Apr 9, 2024 · 7 comments
Assignees
@kou
Labels
Component: C++ Type: enhancement
Milestone
17.0.0

Comments

@kou
Copy link
Member

kou commented Apr 9, 2024 •
edited
Loading

Describe the enhancement requested

[ RUN      ] TestAzureHierarchicalNSGeneric.CopyFile
/home/kou/work/cpp/arrow.kou/cpp/src/arrow/filesystem/test_util.cc:531: Failure
Failed
'fs->CopyFile("AB/abc", "def")' failed with IOError: Failed to copy a blob. (https://XXX.blob.core.windows.net/a2giu5a9ij1szn7tsx4l0z6194h5zqww/AB/abc -> https://XXX.blob.core.windows.net/a2giu5a9ij1szn7tsx4l0z6194h5zqww/def) Azure Error: [CannotVerifyCopySource] 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:2c08ee06-f01e-0062-12d2-807ca0000000
Time:2024-03-28T05:40:28.1650700Z
Request ID: 2c08ee06-f01e-0062-12d2-807ca0000000
[  FAILED  ] TestAzureHierarchicalNSGeneric.CopyFile (1007 ms)
[       OK ] TestAzureHierarchicalNSGeneric.OpenInputFileAsync (394 ms)
[ RUN      ] TestAzureHierarchicalNSGeneric.SpecialChars
/home/kou/work/cpp/arrow.kou/cpp/src/arrow/filesystem/test_util.cc:1178: Failure
Failed
'fs->CopyFile("Blank Char/Special%Char.txt", "Special and%different.txt")' failed with IOError: Failed to copy a blob. (https://XXX.blob.core.windows.net/armiaz82i30wliew7324v0l421a4uhop/Blank%20Char/Special%25Char.txt -> https://XXX.blob.core.windows.net/armiaz82i30wliew7324v0l421a4uhop/Special%20and%25different.txt) Azure Error: [CannotVerifyCopySource] 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:dbfb99e1-801e-008c-4ad2-80d689000000
Time:2024-03-28T05:40:38.3406267Z
Request ID: dbfb99e1-801e-008c-4ad2-80d689000000
[  FAILED  ] TestAzureHierarchicalNSGeneric.SpecialChars (475 ms)

Component(s)

C++
This was referenced Apr 9, 2024
Closed
Closed
@kou
Copy link
Member Author

kou commented Apr 10, 2024

https://learn.microsoft.com/en-us/rest/api/storageservices/data-lake-storage-gen2
It seems that Copy API doesn't exist...?

@felipecrv
Copy link
Contributor

felipecrv commented Apr 10, 2024 •
edited
Loading

We use CopyFromUri [1]. The issue here doesn't seem to be HNS, but authorizing the source with the same credentials used to create the arrow::fs::ArrowFileSystem.

Server failed to authenticate the request. Please refer to the information in the www-authenticate header.

What is XXX?

https://XXX.blob.core.windows.net/a2giu5a9ij1szn7tsx4l0z6194h5zqww/AB/abc
->
https://clearcodearrow.blob.core.windows.net/a2giu5a9ij1szn7tsx4l0z6194h5zqww/def

[1]:

  Azure::Response<Models::CopyBlobFromUriResult> BlobClient::CopyFromUri(
      const std::string& sourceUri,
      const CopyBlobFromUriOptions& options,
      const Azure::Core::Context& context) const

@kou
Copy link
Member Author

kou commented Apr 10, 2024

What is XXX?

I wanted to mask the account but I missed one substitution...

@felipecrv
Copy link
Contributor

Can you report the issue to Microsoft? The function is provided by the SDK and it doesn't seem right that it's failing to authorize the access.

@kou
Copy link
Member Author

kou commented Apr 17, 2024

I couldn't find whether should we report this... So I've opened a question issue in Azure/azure-sdk-for-cpp: Azure/azure-sdk-for-cpp#5542

@kou kou changed the title [C++][FS][Azure] CopyFile doesn't work with Azure hierachical namespace support [C++][FS][Azure] CopyFile doesn't work with Azure hierarchical namespace support Apr 18, 2024
kou added a commit to kou/arrow that referenced this issue Apr 18, 2024
@kou
apacheGH-41095: [C++][FS][Azure] Add support for CopyFile with hierar…
2d0b183 
…chical namespace support

We need to add SAS (Shared Access Signatures) token for source URL.
@kou
Copy link
Member Author

kou commented Apr 18, 2024

Azure/azure-sdk-for-cpp#5542 (comment)

it seems the source URL doesn't contain any authentication information.

You can use SAS token to authenticate the copy source.

@github-actions github-actions bot mentioned this issue Apr 18, 2024
Merged
kou added a commit that referenced this issue Apr 22, 2024
@kou
GH-41095: [C++][FS][Azure] Add support for CopyFile with hierarchical…
25bb627 
… namespace support (#41276)

### Rationale for this change

We need to add SAS (Shared Access Signatures) token for source URL.

### What changes are included in this PR?

Generate SAS token for source URL.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: #41095

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
@kou kou added this to the 17.0.0 milestone Apr 22, 2024
@kou
Copy link
Member Author

kou commented Apr 22, 2024

Issue resolved by pull request 41276
#41276

@kou kou closed this as completed Apr 22, 2024
tolleybot pushed a commit to tmct/arrow that referenced this issue May 2, 2024
@kou @tolleybot
apacheGH-41095: [C++][FS][Azure] Add support for CopyFile with hierar…
10ae8cb 
…chical namespace support (apache#41276)

### Rationale for this change

We need to add SAS (Shared Access Signatures) token for source URL.

### What changes are included in this PR?

Generate SAS token for source URL.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#41095

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
tolleybot pushed a commit to tmct/arrow that referenced this issue May 4, 2024
@kou @tolleybot
apacheGH-41095: [C++][FS][Azure] Add support for CopyFile with hierar…
2e1922b 
…chical namespace support (apache#41276)

### Rationale for this change

We need to add SAS (Shared Access Signatures) token for source URL.

### What changes are included in this PR?

Generate SAS token for source URL.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#41095

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
rok pushed a commit to tmct/arrow that referenced this issue May 8, 2024
@kou @rok
apacheGH-41095: [C++][FS][Azure] Add support for CopyFile with hierar…
6982c92 
…chical namespace support (apache#41276)

### Rationale for this change

We need to add SAS (Shared Access Signatures) token for source URL.

### What changes are included in this PR?

Generate SAS token for source URL.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#41095

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
rok pushed a commit to tmct/arrow that referenced this issue May 8, 2024
@kou @rok
apacheGH-41095: [C++][FS][Azure] Add support for CopyFile with hierar…
b098c97 
…chical namespace support (apache#41276)

### Rationale for this change

We need to add SAS (Shared Access Signatures) token for source URL.

### What changes are included in this PR?

Generate SAS token for source URL.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#41095

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
vibhatha pushed a commit to vibhatha/arrow that referenced this issue May 25, 2024
@kou @vibhatha
apacheGH-41095: [C++][FS][Azure] Add support for CopyFile with hierar…
c421398 
…chical namespace support (apache#41276)

### Rationale for this change

We need to add SAS (Shared Access Signatures) token for source URL.

### What changes are included in this PR?

Generate SAS token for source URL.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#41095

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kou kou

Labels
Component: C++ Type: enhancement
Projects
None yet
Milestone
17.0.0
Development

No branches or pull requests
2 participants
@kou @felipecrv