http_server source should be able to store all headers

[sonnens]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

the http_server source currently only stores a set of predefined headers, it should be able to store all incoming headers or a set of headers minus a defined redacted set

The specific use case for us that we encountered is that Github's webhook notifications store the event type (eg push, pull request, issue, etc) in the X-Github-Event header. Other services store other meaningful information in other headers, etc.

Attempted Solutions

we can try to pre-define all the headers we might encounter , but if we start consuming data from another http source that also stores meaningful information in the headers that information is lost until we 1) realize it and 2) update our vector config to store it in an ever-growing list of mostly null values

Proposal

with source namespacing , vector can choose to store all the headers (or all minus a redacted list) in a headers namespace metadata field & it can be processed further down the line or at the terminus sink.

I've opened a pull request that does this: #18922

References

No response

Version

No response

[erlangparasu]

any updates?