[BUG] S3 Source fails to load all records for large compressed newline-delimited logs

[dlvenable]

Describe the bug

Large log files in S3 with gzip compression do not have all records come through.

One example log file (from Application Load Balancer) has about 80k lines in it. However, Data Prepper only reports about 2000 of these.

[dlvenable]

From testing, it appears that large JSON files do process all lines. And uncompressed log files also process all lines. So this appears to be isolated to compressed files with the newline: codec.

[dlvenable]

From our investigation it appears that the Java GZIPInputStream does not support concatenated gzip files.

See the documentation for GzipCompressorInputStream at https://commons.apache.org/proper/commons-compress/javadocs/api-1.21/index.html for information on the problem with Java's implementation.

See the following post for an explanation of the file structure. https://stackoverflow.com/a/8005155/650176