[C++][FS][Azure] CopyFile doesn't work with Azure hierarchical namespace support

[kou]

Describe the enhancement requested

[ RUN      ] TestAzureHierarchicalNSGeneric.CopyFile
/home/kou/work/cpp/arrow.kou/cpp/src/arrow/filesystem/test_util.cc:531: Failure
Failed
'fs->CopyFile("AB/abc", "def")' failed with IOError: Failed to copy a blob. (https://XXX.blob.core.windows.net/a2giu5a9ij1szn7tsx4l0z6194h5zqww/AB/abc -> https://XXX.blob.core.windows.net/a2giu5a9ij1szn7tsx4l0z6194h5zqww/def) Azure Error: [CannotVerifyCopySource] 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:2c08ee06-f01e-0062-12d2-807ca0000000
Time:2024-03-28T05:40:28.1650700Z
Request ID: 2c08ee06-f01e-0062-12d2-807ca0000000
[  FAILED  ] TestAzureHierarchicalNSGeneric.CopyFile (1007 ms)
[       OK ] TestAzureHierarchicalNSGeneric.OpenInputFileAsync (394 ms)
[ RUN      ] TestAzureHierarchicalNSGeneric.SpecialChars
/home/kou/work/cpp/arrow.kou/cpp/src/arrow/filesystem/test_util.cc:1178: Failure
Failed
'fs->CopyFile("Blank Char/Special%Char.txt", "Special and%different.txt")' failed with IOError: Failed to copy a blob. (https://XXX.blob.core.windows.net/armiaz82i30wliew7324v0l421a4uhop/Blank%20Char/Special%25Char.txt -> https://XXX.blob.core.windows.net/armiaz82i30wliew7324v0l421a4uhop/Special%20and%25different.txt) Azure Error: [CannotVerifyCopySource] 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:dbfb99e1-801e-008c-4ad2-80d689000000
Time:2024-03-28T05:40:38.3406267Z
Request ID: dbfb99e1-801e-008c-4ad2-80d689000000
[  FAILED  ] TestAzureHierarchicalNSGeneric.SpecialChars (475 ms)

Component(s)

C++

[kou]

https://learn.microsoft.com/en-us/rest/api/storageservices/data-lake-storage-gen2
It seems that Copy API doesn't exist...?

[felipecrv]

We use CopyFromUri [1]. The issue here doesn't seem to be HNS, but authorizing the source with the same credentials used to create the arrow::fs::ArrowFileSystem.

Server failed to authenticate the request. Please refer to the information in the www-authenticate header.

What is XXX?

https://XXX.blob.core.windows.net/a2giu5a9ij1szn7tsx4l0z6194h5zqww/AB/abc
->
https://clearcodearrow.blob.core.windows.net/a2giu5a9ij1szn7tsx4l0z6194h5zqww/def

[1]:

  Azure::Response<Models::CopyBlobFromUriResult> BlobClient::CopyFromUri(
      const std::string& sourceUri,
      const CopyBlobFromUriOptions& options,
      const Azure::Core::Context& context) const

[kou]

What is XXX?

I wanted to mask the account but I missed one substitution...

[felipecrv]

Can you report the issue to Microsoft? The function is provided by the SDK and it doesn't seem right that it's failing to authorize the access.

[kou]

I couldn't find whether should we report this... So I've opened a question issue in Azure/azure-sdk-for-cpp: Azure/azure-sdk-for-cpp#5542

[kou]

Azure/azure-sdk-for-cpp#5542 (comment)

it seems the source URL doesn't contain any authentication information.

You can use SAS token to authenticate the copy source.