Inverse Metric-to-log transform

[drdrsh]

Metric-to-log transformer takes in metrics and convert them into log events whereas a Log-to-metric transform is only good for extracting a specific type of metrics from a log event. I am looking for a transform that is just the inverse of metric-to-log which can take in a stream of log events (representing serialized metrics) and convert them back into metrics.

[jszwedko]

Hi @drdrsh !

Can you share a bit more about your use-case? It sounds somewhat like you want to serialize metrics somewhere to transmit them between two Vector instances. Assuming that is the case, rather than using the metric_to_log transform, you can use the native_json codec (or native for protobuf) on sinks to write out all event types in a way that another Vector instance could read them (also using the native_json codec on the source). Would that work for you?

[jszwedko]

Closing since I'm relatively confident that using the native codecs will support your use-case, but feel free to reopen if i'm wrong!

[ee07b415]

Hi @drdrsh, how did you solve this problem? I think I run into the same requirement as you did, our situation is we are using vector act as the metrics collector in one cluster:
sources: avalanche-0: type: prometheus_scrape endpoints: - http://10.0.0.51:9001/metrics namespace: avalanche scrape_interval_secs: 15 sinks: vector_sink: type: http inputs: - avalanche-0 uri: xxx method: post request: headers: auth:xxx

In another cluster we have a http_server source type vector collecting all the requests from the vectors as above to want to aggregate the metrics and send to a central storage place, for example a prometheus server, though we didn't change the metrics data all the way to the server, then the problem is the http_server source type output is log, it seems it did the metrics-to-log transform here, and we find it hard to do the log-to-metrics since we have all 4 types of the metrics and we have to write much more complicated logic to transform into metrics, this use a lot compute resources and can't automatically get the new metrics if they have new tag or field. I think this is the what the inverse mean in the title

[jszwedko]

Hi @ee07b415 !

You should be able to use the native_json codec with the http sink and http_server source to send data over losslessly. However, there is a bug blocking this at the moment: #17056

[ee07b415]

Thanks @jszwedko , the link to the issue is great! I will subscribe to the PR and wait for the updates!