Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: offline encryption for data put #722

Open
chaetognatha opened this issue Oct 25, 2024 · 0 comments
Open

Feature request: offline encryption for data put #722

chaetognatha opened this issue Oct 25, 2024 · 0 comments

Comments

@chaetognatha
Copy link
Collaborator

Problem:

We can only access our storage indirectly via sshfs where we can run dds and in that place we have limited storage.

Starting dds data put on the sshfs mounted network storage means the data has to be read and written over the network several times before it is finally sent, which puts a lot of load on our network and our system and slows things down.

We would need considerable amounts of storage on the edge server that mounts the hpc storage to do the packaging on the local disk.

Proposed solution:

If a few functions can be imported and isolated into a script that can prepare an archive that should be streamed to the cloud storage and dds data put can get a flag that tells it that the data is already prepared offline it could proceed with sending passing the data to the storage endpoint.

I am willing to try to put together a PR for this if I can be pointed in the right direction and get some feedback on how this functionality aligns with the goals of the team!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant