cdc: compress ndjson files for cloud sink

[piyush-singh]

Is your feature request related to a problem? Please describe.
After setting up CDC on the registration cluster, we noticed that the files sent to the S3 bucket were uncompressed and therefore consumed much more space than the logical size in the cluster. For reference, the stmtstats table was 250 GiB in the admin UI, but it was around 8 TB in S3.

We should compress the CDC output files. cc @ajwerner

[ajwerner]

I think that the steps to implement this would require adding:

1. A new option for the cloudstorage sink to indicate the desire for compression
2. A new file suffix to indicate that the data is compressed. Assuming we'd use gzip compression the suiffix would be .gz
3. Actually writing compressed files. I suspect that the compression should occur while bufferring data in memory:

cockroach/pkg/ccl/changefeedccl/sink_cloudstorage.go

Line 52 in beb69e0

buf bytes.Buffer

Probably the solution should store an io.WriteCloser that either points directly to &buf or to a https://golang.org/pkg/compress/gzip/#Writer that wraps the buffer.

This straightforward task adds a lot of value to the cloudstorage sink.

One consideration should be the tolerance of consumers of these files to compression. For example, will snowflake accept compressed files? How painful will the compression make these files to use? @piyush-singh

[piyush-singh]

Snowflake supports creating custom file formats for ingestion which includes specifying compression types. The compression types they support are:

COMPRESSION = AUTO | GZIP | BZ2 | BROTLI | ZSTD | DEFLATE | RAW_DEFLATE | NONE

Having run through this process, this is a fairly trivial change. It would just take a one time, few minute long setup.

[ajwerner]

Snowflake supports creating custom file formats for ingestion which includes specifying compression types.

Awesome! I marked this as good first issue as I think it's easy with high reward. Let's try to get this done for 20.1 one way or another.

[picklerick0496]

@ajwerner Can I work on this issue.?

[ajwerner]

@ajwerner Can I work on this issue.?

Yes you may! Let me know if you run in to any issues.