Feature: Add s3:// as a supported storage root for batch files.

[jamielxcarter]

Feature scope

Targets (data type handling, batching, SQL object generation, etc.)

Description

Extend the Batch feature to support AWS s3 as a supported storage root.

Other changes that would be good to ship with this feature:

• Add cleanup_files=true|false option for Batch config on the target side. False is currently the default behavior for Batch messages. Making it configurable would allow users flexibility on whether they want to keep the batch files after processing them.

[aaronsteers]

@jamielxcarter - Is this a feature you might be interested in contributing?

I just spoke with @edgarrmondragon, and at minimum you'd want to add https://github.com/PyFilesystem/s3fs as a (probably optional or 'extra') python dependency, plus adding some tests.

One possible implementation is to assume that the S3/AWS creds are already discoverable via the underlying S3 libraries (via AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_DEFAULT_REGION, or another method) and then just pass the S3 path to the target using the same convention.

The underlying PyFilesystem library does have S3 capabilities - so in theory the minimum implementation just need to auth successfully and have the extra libraries installed. However, if additional args or configs are needing to be sent, then we'd probably need to iterate a bit on how to send those optional args.

[jamielxcarter]

@aaronsteers - Yes very much so. I'll open up a PR and start contributing when I can! Thanks for the guidance, I'll start there.