Add Keep sink

[talboren]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

People that use Keep (https://www.github.com/keephq/keep) often want to send events from existing services to our service (cloud or self-hosted). The end goal is having an easy-to-configure log sink for Keep to send events to.

Attempted Solutions

No response

Proposal

Add an Keep log sink. I imagine it to be similar to the HTTP implementation (or even use the existing)

An example vector.toml could look like this:

sinks:
  keep:
    inputs:
      - demo_logs
    api_url: http://keep-backend:8080
    api_key: KEEP_API_KEY
    type: keep
    encoding:
      codec: json

References

No response

Version

No response

[sainad2222]

I want to give this a try and wanted to understand few more things about keep after going through their demo_app

1. Do we need a vector provider support at keep side?
2. If not, which provider suits more?

[talboren]

I want to give this a try and wanted to understand few more things about keep after going through their demo_app

1. Do we need a vector provider support at keep side?
2. If not, which provider suits more?

Hey @sainad2222 :) I'm the maintainer of Keep and I think we already discussed some of the items in our Slack.
Happy to help with any open questions you may have

[sainad2222]

@jszwedko Trying to understand what qualifies as a new sink when it can be configured via http too. From what I understood from the examples sinks are present for those who have existing rust sdks/libraries, if they have grpc clients, complex authentication or other things that is hard to support by http. In the example of Keep, we can also use http that can still work

sources:
  sample_prom:
    type: prometheus_scrape
    endpoints:
      - http://localhost:9090/metrics

transforms:
  convert_event:
    inputs:
      - sample_prom
    type: "metric_to_log"
  restructure_event:
    inputs:
      - convert_event
    source: |-
      .event, _err = merge({}, .)
      template = {}
      template.event = .event
      . = template
      .source_type = "prometheus"
    type: "remap"

sinks:
  keep:
    type: "http"
    inputs: ["restructure_event"]
    uri: "http://localhost:8080/alerts/event/vectordev?provider_id=test"
    encoding:
      codec: "json"
    request:
      headers:
        x-api-key: "keepappkey"
        content-type: "application/json"

Will having a dedicated sink help in this case?

[jszwedko]

From what I understood from the examples sinks are present for those who have existing rust sdks/libraries, if they have grpc clients, complex authentication or other things that is hard to support by http.

You have it right. It's a balance but generally we look at how hard it is to reuse an existing sink to determine whether it makes sense to introduce a new one. In this case, it seems fairly straightforward to use the http sink so rather than adding a new one we could just document how to configure the HTTP sink to push to Keep; however, adding a new sink that is just a wrapper around the HTTP sink is pretty straight-forward, too, though and has the advantage of discoverability and slightly better UX.

[sainad2222]

I'll try adding a sink and see if we can remove the additional transformers atleast.

[sainad2222]

For anyone landing on this issue and wants to configure Keep as sink we added support for Vector as a provider in Keep.
Refer this till Keep is added as a sink in Vector.