add Format: JSON Stream support

[tqangxl]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Windows event log to VictoriaLogs
winlogbeat+Vector+VictoriaLogs

Vector：config
[sources.source_winlog_event]
type = "exec"
command = ["D:\Lib\Logs\1.winlogbeat-8.10.4-windows-x86_64\winlogbeat-8.10.4-windows-x86_64\winlogbeat.exe", "-c", "D:\Lib\Logs\1.winlogbeat-8.10.4-windows-x86_64\winlogbeat-8.10.4-windows-x86_64\winlogbeat-My.yml"]
mode = "streaming"

but：
https://docs.victoriametrics.com/VictoriaLogs/data-ingestion/#log-collectors-and-data-ingestion-formats

Log collectors and data ingestion formats

Here is the list of log collectors and their ingestion formats supported by VictoriaLogs:

How to setup the collector	Format: Elasticsearch	Format: JSON Stream	Format: Loki
Filebeat	Yes	No	No
Fluentbit	No	Yes	Yes
Logstash	Yes	No	No
Vector	Yes	No	Yes
Promtail	No	No	Yes

Log collectors and data ingestion formats Here is the list of log collectors and their ingestion formats supported by VictoriaLogs:

How to setup the collector Format: Elasticsearch Format: JSON Stream Format: Loki
Filebeat Yes No No
Fluentbit No Yes Yes
Logstash Yes No No
Vector Yes No Yes
Promtail No No Yes

Attempted Solutions

No response

Proposal

No response

References

No response

Version

No response

[neuronull]

Hello,

Are you using the http sink?

I see from the linked spec

VictoriaLogs accepts JSON line stream aka ndjson at http://localhost:9428/insert/jsonline endpoint.

Have you tried using the http sink with the json encoder, (https://vector.dev/docs/reference/configuration/sinks/http/#encoding.codec)

and the newline_delimited framing method? (https://vector.dev/docs/reference/configuration/sinks/http/#framing.method)

[hagen1778]

Thanks @neuronull @tqangxl !

I can confirm the following config works. Added example for configuring Vector to use VictoriaMetrics Json stream API here.