Push to RIA storage-only remote from mac to linux fails

[ddeepwell]

What is the problem?

Pushing data to a storage-only RIA remote fails when the remote is linux and the local repo is on a mac.

What steps will reproduce the problem?

datalad create test_to_storage_only_ria
cd test_to_storage_only_ria
head -c 1000 /dev/urandom > dummy.bin
datalad save -m "Add annex file"
datalad create-sibling-ria -s arc --alias test --shared group --group rsws --storage-sibling only --new-store-ok ria+ssh://arc.ucalgary.ca:/path/to/storage
datalad push --to arc

Here's the error message:

CommandError: 'git -c diff.ignoreSubmodules=none annex copy --batch -z --to arc --fast --json --json-error-messages --json-progress -c annex.dotfiles=true' failed with exitcode 1 under /private/tmp/test_storage_only_ria [info keys: stdout_json]
> to arc...
Unable to remove /path/to/storage/897/03ee8-379e-4f75-ad4c-41ab8271650c/ora-remote-cc3d0f15-88df-4c1e-b26a-8ba3f0634190/transfer/MD5E-s1000--9fa75b5852530a249e1b83c6e0e7d21d.bin or to obtain write permission in parent directory. -caused by- stat -f%Dp /path/to/storage/897/03ee8-379e-4f75-ad4c-41ab8271650c/ora-remote-cc3d0f15-88df-4c1e-b26a-8ba3f0634190/transfer failed:
This could have failed because --fast is enabled.

DataLad information

datalad version 0.18.3
python 3.10.8

system

• distribution: darwin/22.3.0 13.2.1/arm64
• encoding:
  • default: utf-8
  • filesystem: utf-8
  • locale.prefered: UTF-8
• max_path_length: 282
• name: Darwin
• release: 22.3.0
• type: posix
• version: Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:35 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T8103

Additional context

May be related to #39

Note that the remote is linux while the local machine is macOS. The error message shows that the stat command fails. Running stat -f%Dp file.txt on the local (mac) works, but fails on the remote (linux). Could the wrong command be sent to the remote? Running the steps listed above on linux to a linux remote works.

Have you had any success using DataLad before?

Becoming fairly proficient, having used it steadily over the last year

[yarikoptic]

hm, doesn't it look like datalad/datalad#7296 which as mysteriously came up to our CI runs and then was gone?

[ddeepwell]

Based on the error message from datalad/datalad#7296, it does look similar, but for different reasons. In that issue, it looked like rm was the issue, whereas it is stat here.

[ddeepwell]

Just in case it wasn't git-annex, I upgraded my somewhat old version of git-annex (8.20210631-g9905ec19a) to the latest (10.20230329), to no effect.

[bpoldrack]

Could the wrong command be sent to the remote?

Yes, I suppose that's the issue. Remote execution needs its own detection what system it is running on.

[jsheunis]

I can confirm that this issue reproduces for me with the following setup:

• Dataset: macOS Big Sur 11.7.6
• RIA sibling: Linux server (juseless)

$ datalad create-sibling-ria -s arc --alias test --shared group --group rsws --storage-sibling only --new-store-ok ria+ssh://juseless.inm7.de:/<path-to-ria-location>

[INFO   ] create storage sibling 'arc' ...
create-sibling-ria(ok): /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria (dataset)

$ datalad push --to arc

Transfer data to 'arc':  50%|████████████████████████████████████████████▌                                            | 2.00/4.00 [00:00<00:00, 5.88k Steps/s
CommandError: 'git -c diff.ignoreSubmodules=none annex copy --batch -z --to arc --fast --json --json-error-messages --json-progress -c annex.dotfiles=true' failed with exitcode 1 under /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria [info keys: stdout_json]
> to arc...
  Unable to remove /<path-to-ria-location>/345/4b3bc-8bb6-4de5-9ed4-6b516031d14f/ora-remote-a22c7b35-11fd-498d-b4f6-cb14b0ffbe52/transfer/MD5E-s1000--fc8e14ed76e9387f7408ef8a71cb2aca.bin or to obtain write permission in parent directory. -caused by- stat -f%Dp /<path-to-ria-location>/345/4b3bc-8bb6-4de5-9ed4-6b516031d14f/ora-remote-a22c7b35-11fd-498d-b4f6-cb14b0ffbe52/transfer failed:
  This could have failed because --fast is enabled.
copy: 1 failed

Extra info:

datalad wtf

# WTF
## configuration <SENSITIVE, report disabled by configuration>
## datalad
  - version: 0.18.3+25.gc7d31e1d8
## dataset
  - branches:
    - git-annex@c391781
    - main@3704a85
  - id: 3454b3bc-8bb6-4de5-9ed4-6b516031d14f
  - path: /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria
  - repo: AnnexRepo
## dependencies
  - annexremote: 1.6.0
  - boto: 2.49.0
  - cmd:7z: 17.04
  - cmd:annex: 10.20230227
  - cmd:bundled-git: UNKNOWN
  - cmd:git: 2.31.1
  - cmd:ssh: 8.1p1
  - cmd:system-git: 2.31.1
  - cmd:system-ssh: 8.1p1
  - humanize: 4.6.0
  - iso8601: 1.1.0
  - keyring: 23.13.1
  - keyrings.alt: 4.2.0
  - msgpack: 1.0.5
  - platformdirs: 3.5.0
  - requests: 2.30.0
## environment
  - LANG: en_US.UTF-8
  - LANGUAGE: en_US.UTF-8
  - LC_ALL: en_US.UTF-8
  - LC_CTYPE: en_US.UTF-8
  - LC_TERMINAL: iTerm2
  - LC_TERMINAL_VERSION: 3.4.5
  - PATH: /Users/jsheunis/.rbenv/shims:/Users/jsheunis/opt/miniconda3/envs/next2/bin:/Users/jsheunis/opt/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/jsheunis/.dotfiles/bin:/usr/local/sbin:/usr/X11R6/bin:/usr/local/games:/usr/games
## extensions
  - next:
    - description: What is next in DataLad
    - entrypoints:
      - datalad_next.commands.create_sibling_webdav.CreateSiblingWebDAV:
        - class: CreateSiblingWebDAV
        - load_error: None
        - module: datalad_next.commands.create_sibling_webdav
        - names:
          - create-sibling-webdav
      - datalad_next.commands.credentials.Credentials:
        - class: Credentials
        - load_error: None
        - module: datalad_next.commands.credentials
        - names:
      - datalad_next.commands.download.Download:
        - class: Download
        - load_error: None
        - module: datalad_next.commands.download
        - names:
          - download
      - datalad_next.commands.tree.TreeCommand:
        - class: TreeCommand
        - load_error: None
        - module: datalad_next.commands.tree
        - names:
          - tree
    - load_error: None
    - module: datalad_next
    - version: 1.0.0b2+64.g9a68f85
## git-annex
  - build flags:
    - Assistant
    - Webapp
    - Pairing
    - FsEvents
    - TorrentParser
    - MagicMime
    - Benchmark
    - Feeds
    - Testsuite
    - S3
    - WebDAV
  - dependency versions:
    - aws-0.24
    - bloomfilter-2.0.1.0
    - cryptonite-0.30
    - DAV-1.3.4
    - feed-1.3.2.1
    - ghc-9.4.4
    - http-client-0.7.13.1
    - persistent-sqlite-2.13.1.1
    - torrent-10000.1.1
    - uuid-1.3.15
    - yesod-1.6.2.1
  - key/value backends:
    - SHA256E
    - SHA256
    - SHA512E
    - SHA512
    - SHA224E
    - SHA224
    - SHA384E
    - SHA384
    - SHA3_256E
    - SHA3_256
    - SHA3_512E
    - SHA3_512
    - SHA3_224E
    - SHA3_224
    - SHA3_384E
    - SHA3_384
    - SKEIN256E
    - SKEIN256
    - SKEIN512E
    - SKEIN512
    - BLAKE2B256E
    - BLAKE2B256
    - BLAKE2B512E
    - BLAKE2B512
    - BLAKE2B160E
    - BLAKE2B160
    - BLAKE2B224E
    - BLAKE2B224
    - BLAKE2B384E
    - BLAKE2B384
    - BLAKE2BP512E
    - BLAKE2BP512
    - BLAKE2S256E
    - BLAKE2S256
    - BLAKE2S160E
    - BLAKE2S160
    - BLAKE2S224E
    - BLAKE2S224
    - BLAKE2SP256E
    - BLAKE2SP256
    - BLAKE2SP224E
    - BLAKE2SP224
    - SHA1E
    - SHA1
    - MD5E
    - MD5
    - WORM
    - URL
    - X*
  - local repository version: 10
  - operating system: darwin x86_64
  - remote types:
    - git
    - gcrypt
    - p2p
    - S3
    - bup
    - directory
    - rsync
    - web
    - bittorrent
    - webdav
    - adb
    - tahoe
    - glacier
    - ddar
    - git-lfs
    - httpalso
    - borg
    - hook
    - external
  - supported repository versions:
    - 8
    - 9
    - 10
  - upgrade supported from repository versions:
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    - 8
    - 9
    - 10
  - version: 10.20230227
## location
  - path: /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria
  - type: dataset
## python
  - implementation: CPython
  - version: 3.10.11
## system
  - distribution: darwin/20.6.0 10.16/x86_64
  - encoding:
    - default: utf-8
    - filesystem: utf-8
    - locale.prefered: UTF-8
  - filesystem:
    - CWD:
      - path: /Users/jsheunis/Documents/psyinf/Data/test_to_storage_only_ria
    - HOME:
      - path: /Users/jsheunis
    - TMP:
      - path: /var/folders/g7/720mm8ns7hg3d28_3yyzkx_r0000gp/T
  - max_path_length: 318
  - name: Darwin
  - release: 20.6.0
  - type: posix
  - version: Darwin Kernel Version 20.6.0: Thu Mar  9 20:39:26 PST 2023; root:xnu-7195.141.49.700.6~1/RELEASE_X86_64