New rabbitmq sink

[f2zubac]

Hi Team

In order to be able to completely remove logstash and introduce Vector, it would be nice to introduce RabbitMQ sink plugin.

Sincerely,
Dragan

[binarylogic]

Thanks @f2zubac! Appreciate the detail on your use case. We'll update this issue as we make progress.

[AlyHKafoury]

Hello @binarylogic can I tackle this one, and can you please give me a little guidance where to start ?!

[binarylogic]

Hi @AlyHKafoury absolutely! We'd love to have you contribute. @LucioFranco, since you worked with him on #960 maybe you can recommend the best way to get started?

I also opened the above issue to make this easier in the future.

[LucioFranco]

Hi @AlyHKafoury!

I would suggest taking a look at a couple of our sources mainly the kafka sinks and some of our AWS sinks. They should show how we usually setup our sinks. At a high level they all implement futures::Sink and some use tower_service to use add rate limiting. I believe for this sink we may want to follow the pure futures::Sink method which is shown in kafka. I would suggest using the https://docs.rs/lapin-futures/0.28.0/lapin_futures/ library for interacting with rabitmq. Let me know if you have any questions!

[AlyHKafoury]

@LucioFranco Thank you, I will get started in a couple of days and ask you if I have further questions.

[AlyHKafoury]

@LucioFranco As I understand every sink has to implement an Acker is that correct ? that is the way it communicates it has already sent its buffer of items

[LucioFranco]

@AlyHKafoury you should just have to ack the event once its been sent. So each event will get an ack id you can then ack that event once rabitmq has acked you back. https://github.com/timberio/vector/blob/master/src/sinks/kafka.rs#L195

[njasm]

Hi, is this still being worked on, or is it blocked?
I would like to work on trying to landing a minimum viable rabbitmq sink if no one is working on it

[binarylogic]

@njasm go for it! We're not currently working on it.

[njasm]

@binarylogic ok great, I'm on it! 👍

[theyough]

@njasm any updates ?

[njasm]

@theyough I didn't have the chance to start it, I had to shift focus to other priorities and right now unfortunately I don't have the bandwidth to work on the amqp sink in the near future. - I expect to return to it (if in the meantime no one take a stab at it) but I won't be able to look into it in the next 2/3 months for sure.

[theyough]

#7120, it looks like it s almost done