Write files as Parquet (ideally to S3)

[jvilhuber]

Is your feature request related to a problem? Please describe.

Parquet is a pretty popular format for data analysis and I have a need to transform some json-formatted logs to parquet and into s3 for further consumption by other services/projects. Fluentd seems to support this via the s3 plugin: https://github.com/fluent/fluent-plugin-s3/blob/master/docs/output.md#store_as

Would be very handy if fluent-bit could do this.

Describe the solution you'd like

I would like to be able to specify that the output file format is parquet (optionally with some parquet parameters like compression). It's fine to assume that field-names and field-types match the incoming json.

Describe alternatives you've considered

Alternatives are:

1. use an aws lambda to monitor new s3 files and convert them from json to parquet
2. use File or NATS (or TCP) output modules to pass the data to some other (local) service to convert the files and have those services push to s3.

All those incur the extra complexity of additional components I have to write, monitor, keep up to date, etc.

Additional context

[cosmo0920]

We already have a PR to write parquet with using columnify which is like as Fluentd's stuffs: #8837
We're just waiting for AWS people's responses.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.