http_scrape source should support compression.

[neuronull]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

The http_scrape scrape source implemented in #13793 , does not currently support compression.

[zamazan4ik]

@neuronull as far as I understand, Vector right now uses hyper under the hood as an HTTP client. Hyper does not have the ability to transparently decompress the payload based on Content-Encoding header, but reqwest does.

I see here the following ways to resolve the issue:

• Use reqwest instead of hyper to have a transparent decoding functionality. Could be difficult to migrate from hyper , though.
• Implement Content-Encoding-based decompression on our own. Could be quite tricky since the server and the client could support multiple compression algorithms and they could negotiate somehow the algorithm (honestly, idk much about this stuff).
• Implement compression option as usually we do for other sources/sinks and specify the only accepted algorithm on Vector side (or a couple of them, which we could try one by one - but could be overengineering, though). Not so HTTP-native way, you know

Do you have any other ideas?

[jszwedko]

I think we should do:

• Implement Content-Encoding-based decompression on our own. Could be quite tricky since the server and the client could support multiple compression algorithms and they could negotiate somehow the algorithm (honestly, idk much about this stuff).

The negotiation, I think, is pretty simple. The client sets Accept-Encodings and the server picks one. Vector would just set Accept-Encodings to the set of encodings it can handle.