AWS Liam

Liam is how we decided to call the implementation we did for our events based communication between different Ruby on Rails apps within our AWS infrastructure. We think it can be usefull for some of you that are struggling with this kind of issues too.

We called it Liam in honor to Liam Neeson for so many reasons, being the most important this iconic scene when Bryan (Liam) called Marko and said:

I DONT KNOW WHO YOU ARE, I DON’T KNOW WHAT YOU WANT, BUT I WILL FIND YOU AND I WILL KILL YOU...

We wrote an article at Medium where we explained the whole problem we had when we tried to use plain POST http requests between applications.

After implementing this solution in a couple of apps, Alexis and Sebastian decided to create this gem to encapsulate all the needed code.

The Gem

With AWS Liam we can easily send an event (a message) from App A to App B using SQS and SNS.

The architecture you'll need is very simple: create as many SQS queues as applications you have (it's a good idea to call them liam_NAME_OF_APPLICATION, for example liam_service_a and liam_service_b) and as many SNS topics as events you want to work on (it's a good idea to call them liam_NAME_OF_THE_EVENT, for example liam_ArticleCreated, liam_product_edited, etc) and then subscribe the queues from the Services that need to consume the events from each topic as showed in the image bellow.

Installation

Add this line to your application's Gemfile:

  gem 'aws-liam', require: 'liam'

Then run a simple:

  $ bundle install

Then execute this little rake tast that will generate some configuration files needed.

  $ bundle exec rails g liam:install

2 files will be created:

1. config/liam.yaml
2. lib/task/liam.rake

Go to the first one an setup your credentials and topic endpoints at AWS. You'll need to add here all the events you want to use later.

The second file corresponds to a rake task that will be needed only in the Services that want to be listening for new messages. If your Service will only publish messages (aka will be just a producer, not a consumer) then you can forget (or even delete) this rake task.

Configuration

Liam will fail early if the key for your environment is missing on config/liam.yaml. If you'd like to skip sending messages while you finish your setup, you can add the skip: true option.

For example:

staging:
  <<: *default
  skip: true

Usage

The Producer (Service A)

Every time something happens in Service A that needs to be shared to other applications (for example, an article was published) you need to put this simple three lines (basically create the article JSON, define where do you want to send the message and send the message):

message    = { id: id, title: title, created_at: created_at }
topic_name = 'liam_ArticleCreated'

Liam::Producer.message(topic: topic_name, message: message)

Now we have to define what liam_ArticleCreated means in terms of SNS. For this, we need to go to the config/liam.yaml file that was automatically generated by this gem and add this new event:

...
events:
  liam_ArticleCreated: "arn:aws:sns:us-east-1:xxxxxxxx:liam_ArticleCreated"

Of course, you need to create the topic called liam_ArticleCreated (and make sure that you have access to publish messages there from your Service A).

The Consumer (Service B)

In the oher hand you will have to create a method in Service B that will consume the message received from Service A. At this point Class names is very important.

If you called the topic liam_ArticleCreated then you'll need to create a class called ArticleCreated within an Liam module at Service B, like the following example.

# app/services/liam/article_created.rb

module Liam
  class ArticleCreated
    def initialize(message)
      @message = message
    end

    def process
      # Do all you need to do related to this article (at @message) that has been created at Service A
    end
  end
end

All of these files should live at app/services/liam.

Now you have to run the included task inside the Consumer App (make sure this task runs for ever):

$ bundle exec rake liam:consumer:start production

And that's it!

Testing

You can use the docker-compose.localstack.yml file to leave localstack running if you don't have it installed:

$ docker-compose -f docker-compose.localstack.yml up -d

Before running the test suite you must create the topic we use to test the gem functionality:

$ aws --endpoint-url=http://localhost:4566 sns create-topic --name liam_TestProducer

This is mandatory, otherwise you're going to receive an Aws::SNS::Errors::NotFound: Topic does not exist exception.

After that, you can run the suite with RSpec as usual:

$ bundle exec rspec

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/archdaily/aws-liam.