Support SQS message user attributes #23

piequi · 2023-03-14T09:18:41Z

Release notes

Added support for SQS message user attributes

What does this PR do?

This PR adds the ability to the SQS input plugin to also read and decode SQS user attributes that may be contained in the message.

When some user attributes are defined, the message_attributes key contains a hash of String -> Aws::SQS::Types::MessageAttributeValue with the attributes' name and value.

https://docs.aws.amazon.com/zh_tw/sdk-for-ruby/v3/api/Aws/SQS/Types/MessageAttributeValue.html
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_MessageAttributeValue.html

Why is it important/What is the impact to the user?

Supporting user attributes will allow to enrich events with context metadata when the source emitting the SQS message uses a standard format that cannot be altered.

As mentioned in #22, a Python logging handler does not allow to add cloud metadata to a log record. This could only be achieved by adding user attributes when emitting the log record in a SQS message.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files (and/or docker env variables)
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes Support SQS message (user) attributes #22

Use cases

Screenshots

Logs

cla-checker-service · 2023-03-14T09:18:45Z

💚 CLA has been signed

piequi · 2023-03-15T16:47:05Z

Applying instructions to run tests from https://www.elastic.co/guide/en/logstash/current/contributing-patch-plugin.html simply do not work.

bundle install fails with

Could not find gem 'logstash-devutils' with platform 'x86_64-linux' in rubygems repository https://rubygems.org/ or installed locally.

The source contains the following gems matching 'logstash-devutils':
  * logstash-devutils-0.0.1-java
  * logstash-devutils-0.0.2-java
[...]
  * logstash-devutils-2.4.0-java

Using jruby -S bundle install works

[...]
Using logstash-integration-aws 7.1.1 from source at `.`
Bundle complete! 8 Gemfile dependencies, 82 gems now installed.

but then jruby -S bundle exec rspec fails with

You have requested:
  logstash-devutils >= 0

The bundle currently has logstash-devutils locked at 1.3.6.
Try running `bundle update logstash-devutils`

If you are updating multiple gems in your Gemfile at once,
try passing them all to `bundle update`
Run `bundle install` to install missing gems.

Setting LOGSTASH_PATH environment variable do not change anything; got a hint from

logstash-integration-aws/Gemfile

Line 5 in 183d113

logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"

Looking around I discovered some gradle files and run gradlew assemble && gradlew test. Output is fine but I've no clue if this is running the correct unit tests...

Looking at Travis CI, running jruby -S rake test seems to be the way to go but logstash-devutils is definitely missing...

@logstash-core some help would be much appreciated !

jsvd · 2023-03-15T17:56:50Z

@piequi first of all many thanks for this contribution!!

So here's a set of steps that should get you going:

Step 1: install jruby 9.3 locally

curl https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.3.10.0/jruby-dist-9.3.10.0-bin.tar.gz | tar -zxf - -C /tmp

Step 2: install Logstash

tar -zxf ~/Downloads/logstash-8.6.2-darwin-aarch64.tar.gz -C /tmp

Step 3: setup environment variables

export LOGSTASH_PATH=/tmp/logstash-8.6.2 LOGSTASH_SOURCE=1

Step 4: install bundler, dependencies and vendor jars

/tmp/jruby-9.3.10.0/bin/jruby -S gem install bundler 
/tmp/jruby-9.3.10.0/bin/jruby -S bundle install
/tmp/jruby-9.3.10.0/bin/jruby -S bundle exec rake vendor

Step 5: run tests

/tmp/jruby-9.3.10.0/bin/jruby -S bundle exec rspec

Those docs are in serious need of updating, I'll track this need and hopefully get to it soon.

piequi · 2023-03-20T13:46:45Z

Hi @jsvd !
Thanks for your help !

Following your instructions, everything goes fine until jruby -S bundle exec rake vendor where I get:

[!] There was an error while loading `logstash-core-plugin-api.gemspec`: The logstash-core-api need to be build on jruby. Bundler cannot continue.

 #  from /home/marc-antoine/bin/logstash-8.6.2/logstash-core-plugin-api/logstash-core-plugin-api.gemspec:46
 #  -------------------------------------------
 #    else
 >      raise "The logstash-core-api need to be build on jruby"
 #    end
 #  -------------------------------------------

It seems some commands aren't executed using jruby as I see that :

$ irb
irb(main):001:0> puts RUBY_PLATFORM
x86_64-linux-gnu
=> nil

$ jruby -S irb                    
irb(main):001:0> puts RUBY_PLATFORM
java
=> nil

For info:

$ jruby -v    
jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 11.0.17+8-post-Ubuntu-1ubuntu2 on 11.0.17+8-post-Ubuntu-1ubuntu2 +jit [x86_64-linux]

$ ruby -v                          
ruby 3.0.4p208 (2022-04-12 revision 3fa771dded) [x86_64-linux-gnu]

(Running the exact commands you provided and extracting JRuby to /tmp gives the same ouputs)

piequi · 2023-03-22T22:14:40Z

Here is a Dockerfile highlighting the error I get :

FROM ubuntu:latest

USER root

RUN apt-get update
RUN apt-get install -y curl default-jre git rake

RUN adduser --disabled-password  test

USER test
WORKDIR /home/test

RUN curl https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.3.10.0/jruby-dist-9.3.10.0-bin.tar.gz \
  | tar -zxf -

RUN curl https://artifacts.elastic.co/downloads/logstash/logstash-oss-8.6.2-linux-x86_64.tar.gz \
  | tar -zxf -

ENV LOGSTASH_PATH=/home/test/logstash-8.6.2
ENV LOGSTASH_SOURCE=1

RUN /home/test/jruby-9.3.10.0/bin/jruby -S gem install bundler rspec

RUN git clone https://github.com/logstash-plugins/logstash-integration-aws.git

WORKDIR /home/test/logstash-integration-aws

RUN /home/test/jruby-9.3.10.0/bin/jruby -S bundle install
RUN /home/test/jruby-9.3.10.0/bin/jruby -S bundle exec rake vendor
RUN /home/test/jruby-9.3.10.0/bin/jruby -S bundle exec rspec

jsvd · 2023-03-23T11:24:40Z

Here are a few tweaks:

FROM ubuntu:latest

USER root

RUN apt-get update
RUN apt-get install -y curl default-jre git rake

RUN adduser --disabled-password  test

USER test
WORKDIR /home/test

RUN curl https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.3.10.0/jruby-dist-9.3.10.0-bin.tar.gz \
  | tar -zxf -

ENV PATH="/home/test/jruby-9.3.10.0/bin/:${PATH}"

RUN curl https://artifacts.elastic.co/downloads/logstash/logstash-8.6.2-linux-x86_64.tar.gz \
  | tar -zxf -

ENV LOGSTASH_PATH=/home/test/logstash-8.6.2
ENV LOGSTASH_SOURCE=1

RUN jruby -S gem install bundler rspec

RUN git clone https://github.com/logstash-plugins/logstash-integration-aws.git

WORKDIR /home/test/logstash-integration-aws

RUN bundle install
RUN bundle exec rake vendor
RUN bundle exec rspec

piequi · 2023-03-23T12:54:48Z

Thanks @jsvd !
I'm good to fix existing tests and add a few more to validate my changes !

This reverts commit 3ef86af.

piequi added 3 commits March 14, 2023 09:16

Decode SQS attributes

eb95bae

Update doc

29cbb05

Update change log

941b1a0

Fix styling

c5af410

piequi force-pushed the feat/support-message-user-attributes-EP-11784 branch from bb0cacc to 61239af Compare March 15, 2023 16:33

piequi force-pushed the feat/support-message-user-attributes-EP-11784 branch 2 times, most recently from 2c97e75 to f7fc78e Compare March 20, 2023 21:41

WIP: fix existing unit tests

edc0d6c

piequi force-pushed the feat/support-message-user-attributes-EP-11784 branch from f7fc78e to edc0d6c Compare March 20, 2023 21:49

piequi added 4 commits November 14, 2023 14:02

Pin SQS SDK version to 1.65.0

3ef86af

Merge branch 'main' into feat/support-message-user-attributes-EP-11784

4827d13

Revert "Pin SQS SDK version to 1.65.0"

9a431ad

This reverts commit 3ef86af.

Remove core api pin

64516da

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support SQS message user attributes #23

Support SQS message user attributes #23

piequi commented Mar 14, 2023 •

edited

Loading

cla-checker-service bot commented Mar 14, 2023 •

edited

Loading

piequi commented Mar 15, 2023

jsvd commented Mar 15, 2023

piequi commented Mar 20, 2023

piequi commented Mar 22, 2023

jsvd commented Mar 23, 2023

piequi commented Mar 23, 2023

Support SQS message user attributes #23

Are you sure you want to change the base?

Support SQS message user attributes #23

Conversation

piequi commented Mar 14, 2023 • edited Loading

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service bot commented Mar 14, 2023 • edited Loading

piequi commented Mar 15, 2023

jsvd commented Mar 15, 2023

piequi commented Mar 20, 2023

piequi commented Mar 22, 2023

jsvd commented Mar 23, 2023

piequi commented Mar 23, 2023

piequi commented Mar 14, 2023 •

edited

Loading

cla-checker-service bot commented Mar 14, 2023 •

edited

Loading