Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement I/O from/to S3 #234

Closed
2 tasks done
holtgrewe opened this issue Oct 19, 2023 · 1 comment · Fixed by #239 or #242
Closed
2 tasks done

Implement I/O from/to S3 #234

holtgrewe opened this issue Oct 19, 2023 · 1 comment · Fixed by #239 or #242
Labels
enhancement New feature or request
Milestone

Comments

@holtgrewe
Copy link
Contributor

holtgrewe commented Oct 19, 2023

Is your feature request related to a problem? Please describe.
VarFish shall work on S3 storage internally. For this, the commands should be able to read from and write to S3 storage.

Describe the solution you'd like

  • Interpret the environment variables from below and enable S3 mode in this case.
  • Interpret absolute paths to always come from the local file system.

Describe alternatives you've considered
N/A

Additional context
Environment variables (cf. aws cli docs)

  • AWS_ACCESS_KEY_ID -- must be set, eanbles s3 mode
  • AWS_SECRET_ACCESS_KEY -- must be set
  • AWS_ENDPOINT_URL -- must be set
  • AWS_REGION=us-east-1
@holtgrewe
Copy link
Contributor Author

OK, so direct upload to S3 is not possible as you can only easily stream upload from a file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
1 participant