You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Combining the information in #80 and #98 it seems advisable to abandon the IOBase and friends framework.
The current ORA implementation assumes that there is a finite set of "IO operations" that can be represented by all "protocols". This has worked well for SSH and local access (both actually work with a file system). It has not scaled to HTTP requests. it is unlikely to work for general object stores and other possibilities to store files.
I believe it would be more meaningful to have separate implementations for special remote operations (e.g. STORE, RETRIEVE, ...), rather than have one implementation that call special implementations for a fixed procedure of "filesystem operations". I think the resulting code would be leaner, more readable, and yield a more efficient operation.
The text was updated successfully, but these errors were encountered:
mih
changed the title
Abandon IO abstractions, switch to a :wmulti-flavor ORA remote
Abandon IO abstractions, switch to a multi-flavor ORA remote
Sep 23, 2023
I believe it would be more meaningful to have separate implementations for special remote operations (e.g. STORE, RETRIEVE, ...), rather than have one implementation that call special implementations for a fixed procedure of "filesystem operations". I think the resulting code would be leaner, more readable, and yield a more efficient operation.
I agree with that. Especially if we take into account the support of other storage backends, e.g. s3-buckets. Different storage systems might best be supported by a combination of uncurl-remotes and specific implementations.
What I am currently wondering about are the backward compatibility requirements:
Existing stores must remain accessible with the new implementation and the schemes: ria+file, ria+ssh, and ria+http.
New schemes, e.g. ria+s3, ria+webdav, ria+googledrive, etc. can be supported by any implementation with no necessity for backward compatibility.
Combining the information in #80 and #98 it seems advisable to abandon the
IOBase
and friends framework.The current ORA implementation assumes that there is a finite set of "IO operations" that can be represented by all "protocols". This has worked well for SSH and local access (both actually work with a file system). It has not scaled to HTTP requests. it is unlikely to work for general object stores and other possibilities to store files.
I believe it would be more meaningful to have separate implementations for special remote operations (e.g. STORE, RETRIEVE, ...), rather than have one implementation that call special implementations for a fixed procedure of "filesystem operations". I think the resulting code would be leaner, more readable, and yield a more efficient operation.
The text was updated successfully, but these errors were encountered: