Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AzCopy fails to write xattr #2761

Open
tsvietOK opened this issue Jul 25, 2024 · 10 comments · May be fixed by #2762
Open

AzCopy fails to write xattr #2761

tsvietOK opened this issue Jul 25, 2024 · 10 comments · May be fixed by #2762
Assignees

Comments

@tsvietOK
Copy link

tsvietOK commented Jul 25, 2024

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

10.25.1

Which platform are you using? (ex: Windows, Mac, Linux)

Linux (Ubuntu 22.04 VM in Azure)

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy sync "https://<storageName>.blob.core.windows.net/<container>/<folder>/?<sig>" "." --delete-destination true --compare-hash "MD5" --cap-mbps 400

What problem was encountered?

When trying to sync files from storage account to Ubuntu 22.04 I getting xattr related errors:

2024/07/24 16:15:38 failed to write hash data for edd - Copy (3).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/24 16:15:38 failed to write hash data for edd - Copy.txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/24 16:15:38 failed to write hash data for edd - Copy (2).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode

Important! Even when it shows errors in the log and shows all operations as failed, it downloads files anyway.

How can we reproduce the problem in the simplest way?

  1. Upload test files to storage account
  2. Create Ubuntu 22.04 VM in Azure
  3. Attach data disk
  4. Mount data disk (as ext4)
  5. Sync files from storage account to this data disk using the command azcopy sync (as above)

Have you found a mitigation/solution?

I installed xattr package, but it didn't help.
Tried to use OS agnostic way, but there is another issue with it #2758
The potential reason for this issue is that you are trying to write xattr without "user." prefix. I tried to set xattrs for test file by command xattr -w sss testVal test.xml and it failed with error [Errno 95] Operation not supported: b'test.xml', but it works as xattr -w user.sss testVal test.xml.
As far as I understand, user_xattr is enabled for ext4 filesystem by default. Also, I tried to enable it explicitly (via /etc/fstab).
It is described here: https://stackoverflow.com/questions/60893372/why-must-a-user-be-prepended-to-the-name-when-setting-a-files-xattr-with-os

@tsvietOK
Copy link
Author

tsvietOK commented Jul 25, 2024

Command log:

INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job d0c17a5c-fb5d-c449-503e-38a02477af0f has started
Log file is located at: /home/user/.azcopy/d0c17a5c-fb5d-c449-503e-38a02477af0f.log

INFO: One or more hash storage operations (read/write) have failed. Check the scanning log for details.
INFO: Deleting extra file: edd - Copy (7).txt
INFO: Deleting extra file: edd.txt
INFO: Deleting extra file: edd - Copy (11).txt
INFO: Deleting extra file: edd - Copy (10).txt
INFO: Deleting extra file: edd - Copy (5).txt
INFO: Deleting extra file: edd - Copy (4).txt
INFO: Deleting extra file: 123/ssss.txt
INFO: Deleting extra file: edd - Copy (2).txt
INFO: Deleting extra file: edd - Copy (8).txt
INFO: Deleting extra file: edd - Copy (6).txt
INFO: Deleting extra file: edd - Copy (9).txt
INFO: Deleting extra file: edd - Copy.txt
INFO: Deleting extra file: edd - Copy (3).txt
INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
0.0 %, 0 Done, 3 Failed, 0 Pending, 3 Total, 2-sec Throughput (Mb/s): 268.3674

Job d0c17a5c-fb5d-c449-503e-38a02477af0f Summary
Files Scanned at Source: 3
Files Scanned at Destination: 13
Elapsed Time (Minutes): 0.1002
Number of Copy Transfers for Files: 3
Number of Copy Transfers for Folder Properties: 0
Total Number of Copy Transfers: 3
Number of Copy Transfers Completed: 0
Number of Copy Transfers Failed: 3
Number of Deletions at Destination: 13
Total Number of Bytes Transferred: 0
Total Number of Bytes Enumerated: 270993275
Final Job Status: Failed

Scanning log:

2024/07/25 17:35:56 AzcopyVersion  10.25.1
2024/07/25 17:35:56 OS-Environment  linux
2024/07/25 17:35:56 OS-Architecture  amd64
2024/07/25 17:35:56 Log times are in UTC. Local time is 25 Jul 2024 17:35:56
2024/07/25 17:35:56 failed to read hash data for edd - Copy (11).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (3).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd.txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (2).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy.txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (7).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (5).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (10).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (4).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (8).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (6).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for edd - Copy (9).txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to read hash data for 123/ssss.txt: failed to read xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (11).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (2).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy.txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (5).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (10).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (7).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd.txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (3).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (8).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (6).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (9).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for edd - Copy (4).txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 failed to write hash data for 123/ssss.txt: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/07/25 17:35:56 ==> REQUEST/RESPONSE (Try=1/12.696453ms, OpTime=54.29216ms) -- RESPONSE SUCCESSFULLY RECEIVED
   HEAD https://<storage>.blob.core.windows.net/<container>/<folder>?se=2024-07-26T01%3A35%3A41Z&sig=-REDACTED-&sp=racwdl&spr=https&sr=c&st=2024-07-25T17%3A35%3A41Z&sv=2022-11-02
   X-Ms-Request-Id: [fec348af-301e-0037-0fb9-de42b6000000]

2024/07/25 17:35:56 ==> REQUEST/RESPONSE (Try=1/6.572306ms, OpTime=6.613234ms) -- RESPONSE SUCCESSFULLY RECEIVED
   GET https://<storage>.blob.core.windows.net/<container>?comp=list&delimiter=%2F&include=metadata&prefix=<prefix>%2F&restype=container&se=2024-07-26T01%3A35%3A41Z&sig=-REDACTED-&sp=racwdl&spr=https&sr=c&st=2024-07-25T17%3A35%3A41Z&sv=2022-11-02
   X-Ms-Request-Id: [fec348ba-301e-0037-17b9-de42b6000000]

2024/07/25 17:35:56 Deleting extra file: edd - Copy (7).txt
2024/07/25 17:35:56 Deleting extra file: edd.txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (11).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (10).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (5).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (4).txt
2024/07/25 17:35:56 Deleting extra file: 123/ssss.txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (2).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (8).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (6).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (9).txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy.txt
2024/07/25 17:35:56 Deleting extra file: edd - Copy (3).txt
2024/07/25 17:36:02 Closing Log

@tsvietOK
Copy link
Author

tsvietOK commented Jul 25, 2024

Installed xattr:
$ dpkg -s xattr

Package: xattr
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 17
Maintainer: Ubuntu Developers <[email protected]>
Architecture: amd64
Source: python-xattr
Version: 0.9.7-1build4
Depends: python3:any, python3-xattr (= 0.9.7-1build4), python3-pkg-resources
Description: tool for manipulating filesystem extended attributes
 xattr tool allows manipulation of the filesystem extended attributes present
 in some operating systems (GNU/Linux included).
 .
 This package contains the xattr cli tool.
Homepage: https://github.com/xattr/xattr
Original-Maintainer: Federico Ceratto <[email protected]>

@tsvietOK tsvietOK linked a pull request Jul 26, 2024 that will close this issue
@tsvietOK
Copy link
Author

https://go.googlesource.com/sys.git/+/master/unix/xattr_bsd.go#31

func xattrnamespace(fullattr string) (ns int, attr string, err error) {
  s := strings.IndexByte(fullattr, '.')
  if s == -1 {
    return -1, "", ENOATTR
  }
   
  namespace := fullattr[0:s]
  attr = fullattr[s+1:]
   
  switch namespace {
    case "user":
      return EXTATTR_NAMESPACE_USER, attr, nil
    case "system":
      return EXTATTR_NAMESPACE_SYSTEM, attr, nil
    default:
      return -1, "", ENOATTR
  }
}

@tsvietOK
Copy link
Author

Fix suggested: #2762

@ashruti-msft ashruti-msft self-assigned this Jul 31, 2024
@seanmcc-msft seanmcc-msft removed the bug label Aug 1, 2024
@adreed-msft
Copy link
Member

Hi there. Apologies for the delayed response. Xattr has to be available at a filesystem level, which is usually a flag when mounting the fs.

There are two potential mitigations here

  1. Enable xattr on the target filesystem
  2. Use a different hash storage mode (check the docs for --local-hash-storage-mode and --hash-meta-dir for a dependency agnostic solution)

@tsvietOK
Copy link
Author

tsvietOK commented Aug 5, 2024

As far as I understand, user_xattr is enabled for ext4 filesystem by default.

OS agnostic solution also has its own issue as I described in issue description.

@tsvietOK
Copy link
Author

tsvietOK commented Aug 5, 2024

image

@tsvietOK
Copy link
Author

tsvietOK commented Aug 5, 2024

@adreed-msft I tested my changes from PR #2762 and it works as expected.

@tsvietOK
Copy link
Author

tsvietOK commented Aug 7, 2024

@hmi12
Copy link

hmi12 commented Nov 27, 2024

Got the same....

2024/11/27 10:20:48 failed to write hash data for ._data01: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/11/27 10:20:48 failed to write hash data for sync: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/11/27 10:20:48 failed to write hash data for sync2: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/11/27 10:20:48 failed to write hash data for sync3: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode
2024/11/27 10:20:48 failed to write hash data for sync4: failed to write xattr: operation not supported; consider utilizing an OS-agnostic hash storage mode

Use the XFS filesystem:

/dev/sdb1 on /data type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants