clone: silently stuck on transferkey #194
Comments
yarikoptic
changed the title
|
I can reproduce the issue for the given repo (tried only git-clone). I can also confirm that the repo content key is present and contains a valid repo (downloaded manually to check). I can also confirm that the Bottom line: server-side things look normal. Looking at the clone, and its Rerunning things with full debug: Observation of "odd" things (may or may not be useful clues):
Will continue investigating... |
|
Looks like this is dependent on the sibling mode. "human-readable" exports seemingly work fine... |
|
I improved the error messaging in It fails on a fundamental access authorization level. This is confusing. The With that token, the following line in try-except fails with # next one just sets up the stage, no requests performed yet, hence
# no error checking needed
# supply both auth credentials, so osfclient can fall back on user/pass
# if needed
osf = OSF(**creds)
# next one performs initial auth
try:
self.node = osf.project(node_id)
except Exception as e:
# we need to raise RemoteError() such that PREPARE-FAILURE
# is reported, sadly that doesn't give users any clue
# TODO support datalad logging here
raise RemoteError(
f'Failed to obtain OSF node handle ({node_id!r}): {e!r} [using credentials {creds!r}]'
)Will look into not passing any credential now... And it seems not having or not passing credentials now let's me replicate the "hangs forever" observation. Update: It is not "forever", it is merely very slow. It seems that when not passing credentials, each request literally takes minutes (and more than one request needs to be made). However, the requests eventually succeed! If this assessment is correct, I'd speculate that this is an OSF "bug". They reject credentials due to a mismatch with a target project, even though no credential/authorization is needed. The workaround via no authentication is then soured by a service-side penalty for unauthenticated access. If this is a correct assessment, there is little we can do implementation-wise. If we try unauthenticated access first, in order to enable "public access", we introduce a 10-20min latency for any "personal use" -- unacceptable. The best thing we can do is to document this behavior, and to make it clearer in the logs what is happening, and how to bypass credentials submissing to make things work at all -- even if only at a ridiculous speed. Here is how it looks when not passing a credential: ❯ git clone -v osf://gvjbn osf9
Cloning into 'osf9'...
[DATALAD-ANNEX] Connecting git-upload-pack
[DATALAD-ANNEX] Get refs from remote
[DATALAD-ANNEX] Set mirror to remote state
[DATALAD-ANNEX] (recording state in git...)
[DATALAD-ANNEX] get XDLRA--repo-export (from origin...)
ok
[DATALAD-ANNEX] Extracting repository archive
[DATALAD-ANNEX] Established mirror
remote: Enumerating objects: 52257, done.
remote: Total 52257 (delta 0), reused 0 (delta 0), pack-reused 52257
Receiving objects: 100% (52257/52257), 4.43 MiB | 42.38 MiB/s, done.
Resolving deltas: 100% (6575/6575), done.
git clone -v osf://gvjbn osf9 8.67s user 3.30s system 0% cpu 20:33.28 total
❯ ls osf9
20.2.0 20.2.2 20.2.4 21.0.4 23.0.0
20.2.1 20.2.3 20.2.5 22.1.1 corrupted-template_MNI152NLin2009cAsym_res-01_T1w.tar.gz(as shown in the log, this took 20min) |
@adswa can you confirm (or not) that all "human-readable" projects you tested are "your's", and that you have a token present in your setup? If so, my assessment would hold. If not, something would still be missing. |
|
yes, indeed - all my own projects, and with a token setup |
Originally was done using
datalad clonebut that one was silent, so ran directly with GIT_TRACE to hope to gain some clue, but didn't really, so I Ctrl-C'ed it❯ GIT_TRACE=1 git clone osf://gvjbn
rerunning after complete redo of virtualenv and thus all fresh updates to datalad etc:
❯ DATALAD_LOG_LEVEL=debug GIT_TRACE=1 git clone osf://gvjbn
and there it hangs. So it is an 'annex transferkey' but I do not see any size change there etc... if I make copy/rerun manually with --debug there
full log trace
so seems stalling at that git-annex-remote-osf TRANSFER request silently. and again size of folder doesn't grow.
The text was updated successfully, but these errors were encountered: