feature: Add built-in support for BATCH message type (aka FAST_SYNC spec)

[MeltyBot]

This feature would bring BATCH message type support to SDK taps and targets, beginning with the .jsonl.gz file format saved to local storage.

Spec discussion:

• Spec discussion: BATCH message type, first iteration #859

---

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/9

Originally created by @aaronsteers on 2020-12-18 18:12:08

---

Spec discussion (old)

This enhancement would add framework support for the new FAST_SYNC spec as described on the meltano thread (https://gitlab.com/meltano/meltano/-/issues/2364).

To kick off the discussion, what about this as a strawman spec:

List of spec changes to support Fast Sync (partial, wip):

• register_batch_export_handler() - Registers a handler function to respond to batch export requests. Includes in the registration command a declaration of what file type and storage options are supported by the handler, along with the relative priority of the specific handler.
• register_batch_import_handler() - Same as above but for targets.

Following from other design practices, we would not require that the tap author knows how to implement the BATCH message type, just that they return file paths in a way we can properly pass them to the downstream client (according to spec work on https://gitlab.com/meltano/meltano/-/issues/2364).

Example:

In the case of a Redshift UNLOAD command, the register_batch_export_handler() might give a function to execute the UNLOAD command, save to S3, and then download the files locally and return the corresponding local filepaths.

[MeltyBot]

View 17 previous comments from the original issue on GitLab

[aaronsteers]

Noting from a talk with one user today: 20MM rows seems to be a turning point performance wise.

When we test performance, would be helpful to use 20-50MM rows as performance evaluation framework.

tap-stackoverflow was suggested as a source that can generate this volume of records fairly easily. Or here.

[edgarrmondragon]

I'm making some improvements to documentation and developer workflow that will come in handy for this work:

• bug: Constructor parameters descriptions are not added to generated autodocs #899