Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abandon IO abstractions, switch to a multi-flavor ORA remote #99

Open
mih opened this issue Sep 23, 2023 · 1 comment
Open

Abandon IO abstractions, switch to a multi-flavor ORA remote #99

mih opened this issue Sep 23, 2023 · 1 comment

Comments

@mih
Copy link
Member

mih commented Sep 23, 2023

Combining the information in #80 and #98 it seems advisable to abandon the IOBase and friends framework.

The current ORA implementation assumes that there is a finite set of "IO operations" that can be represented by all "protocols". This has worked well for SSH and local access (both actually work with a file system). It has not scaled to HTTP requests. it is unlikely to work for general object stores and other possibilities to store files.

I believe it would be more meaningful to have separate implementations for special remote operations (e.g. STORE, RETRIEVE, ...), rather than have one implementation that call special implementations for a fixed procedure of "filesystem operations". I think the resulting code would be leaner, more readable, and yield a more efficient operation.

@mih mih changed the title Abandon IO abstractions, switch to a :wmulti-flavor ORA remote Abandon IO abstractions, switch to a multi-flavor ORA remote Sep 23, 2023
@christian-monch
Copy link

christian-monch commented Sep 28, 2023

I believe it would be more meaningful to have separate implementations for special remote operations (e.g. STORE, RETRIEVE, ...), rather than have one implementation that call special implementations for a fixed procedure of "filesystem operations". I think the resulting code would be leaner, more readable, and yield a more efficient operation.

I agree with that. Especially if we take into account the support of other storage backends, e.g. s3-buckets. Different storage systems might best be supported by a combination of uncurl-remotes and specific implementations.

What I am currently wondering about are the backward compatibility requirements:

  1. Existing stores must remain accessible with the new implementation and the schemes: ria+file, ria+ssh, and ria+http.
  2. New schemes, e.g. ria+s3, ria+webdav, ria+googledrive, etc. can be supported by any implementation with no necessity for backward compatibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants