x-pack/filebeat/input/awss3[integration test]Test integration tests on localstack #35727

bhapas · 2023-06-09T09:03:56Z

What does this PR do?

This PR provides support in awss3 input_integration_test to test input against an emulated cloud environment Localstack.

This PR adds an example integration test to test SQS notifications when an Object is PUT in S3 where S3 and SQS are provisioned in Localstack.

This PR adds support to run the test in CI too including cleanup in both positive and negative scenarios.

Why is it important?

This PR provides a lot of flexibility for developers to test awss3 input against emulated cloud env instead of waiting to run on real AWS resources , which saves both time and money.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run docker-compose -f .ci/jobs/docker-compose.yml up -d
Then navigate to x-pack/filebeat/input/awss3
Run go test -tags aws,integration -run TestInputRunSQSOnLocalstack -v

Related issues

Closes [Filebeat S3 input]Provide support to run integration test against localstack #35758

Logs

input/awss3 - (test-queue-localstack) > go test -tags integration,aws -run TestInputRunSQSOnLocalstack -v
=== RUN   TestInputRunSQSOnLocalstack
    input_integration_test.go:206: .gitignore
    input_integration_test.go:206: .terraform
    input_integration_test.go:206: .terraform.lock.hcl
    input_integration_test.go:206: README.md
    input_integration_test.go:206: localstack.tf
    input_integration_test.go:206: main.tf
    input_integration_test.go:206: outputs-localstack.yml
    input_integration_test.go:206: outputs.tf
    input_integration_test.go:206: terraform.tfstate
    input_integration_test.go:206: variables.tf
    input_integration_test.go:440: Drained 1 SQS messages.
    input_integration_test.go:234: File uploaded to http://localhost:4566/filebeat-s3-integtest-localstack-9uoyjj/events-array.json
    input_integration_test.go:234: File uploaded to http://localhost:4566/filebeat-s3-integtest-localstack-9uoyjj/invalid.json
    input_integration_test.go:234: File uploaded to http://localhost:4566/filebeat-s3-integtest-localstack-9uoyjj/log.json
    input_integration_test.go:234: File uploaded to http://localhost:4566/filebeat-s3-integtest-localstack-9uoyjj/log.ndjson
    input_integration_test.go:234: File uploaded to http://localhost:4566/filebeat-s3-integtest-localstack-9uoyjj/multiline.json
    input_integration_test.go:234: File uploaded to http://localhost:4566/filebeat-s3-integtest-localstack-9uoyjj/multiline.json.gz
    input_integration_test.go:234: File uploaded to http://localhost:4566/filebeat-s3-integtest-localstack-9uoyjj/multiline.txt
    input_integration_test.go:234: File uploaded to http://localhost:4566/filebeat-s3-integtest-localstack-9uoyjj/log.txt
{"log.level":"info","@timestamp":"2023-07-19T09:05:14.780+0200","log.logger":"aws-s3","log.origin":{"file.name":"awss3/input.go","file.line":199},"message":"AWS api_timeout is set to 2m0s.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-19T09:05:14.781+0200","log.logger":"aws-s3","log.origin":{"file.name":"awss3/input.go","file.line":200},"message":"AWS region is set to us-east-1.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-19T09:05:14.781+0200","log.logger":"aws-s3","log.origin":{"file.name":"awss3/input.go","file.line":201},"message":"AWS SQS visibility_timeout is set to 5m0s.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-19T09:05:14.781+0200","log.logger":"aws-s3","log.origin":{"file.name":"awss3/input.go","file.line":202},"message":"AWS SQS max_number_of_messages is set to 5.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-19T09:05:14.781+0200","log.logger":"metric_registry","log.origin":{"file.name":"inputmon/input.go","file.line":63},"message":"registering","input_type":"aws-s3","id":"test_id","key":"test_id","uuid":"4a441e83-60d9-4a70-85be-b1f6dc1fba04","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs","log.origin":{"file.name":"awss3/sqs.go","file.line":80},"message":"Received 5 SQS messages.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":303},"message":"SQS message contained 1 S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"0a529b02-ae0f-4a65-bda1-060c65335cb3","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":303},"message":"SQS message contained 1 S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"1a93f553-140e-4450-a90f-fe00ac3a27ce","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":303},"message":"SQS message contained 1 S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"5fafe7f0-4ce4-4b27-a077-68b3819d0c9a","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":303},"message":"SQS message contained 1 S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"ea736dea-fbe6-4b11-a40d-3a025fced148","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":124},"message":"Begin S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"5fafe7f0-4ce4-4b27-a077-68b3819d0c9a","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"invalid.json","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":124},"message":"Begin S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"1a93f553-140e-4450-a90f-fe00ac3a27ce","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"events-array.json","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":303},"message":"SQS message contained 1 S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"295de242-61fa-45c7-809c-89de9175b7b9","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":124},"message":"Begin S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"ea736dea-fbe6-4b11-a40d-3a025fced148","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"multiline.json","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":124},"message":"Begin S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"295de242-61fa-45c7-809c-89de9175b7b9","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"log.json","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.799+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":124},"message":"Begin S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"0a529b02-ae0f-4a65-bda1-060c65335cb3","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"log.ndjson","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.818+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":132},"message":"End S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"ea736dea-fbe6-4b11-a40d-3a025fced148","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"multiline.json","elapsed_time_ns":18290208,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.818+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":348},"message":"End processing SQS S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"ea736dea-fbe6-4b11-a40d-3a025fced148","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.820+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":132},"message":"End S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"1a93f553-140e-4450-a90f-fe00ac3a27ce","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"events-array.json","elapsed_time_ns":20465584,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.820+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":348},"message":"End processing SQS S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"1a93f553-140e-4450-a90f-fe00ac3a27ce","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.821+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":132},"message":"End S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"295de242-61fa-45c7-809c-89de9175b7b9","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"log.json","elapsed_time_ns":22195708,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.821+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":348},"message":"End processing SQS S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"295de242-61fa-45c7-809c-89de9175b7b9","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.822+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":132},"message":"End S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"0a529b02-ae0f-4a65-bda1-060c65335cb3","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"log.ndjson","elapsed_time_ns":22394250,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.822+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":348},"message":"End processing SQS S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"0a529b02-ae0f-4a65-bda1-060c65335cb3","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.823+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":132},"message":"End S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"5fafe7f0-4ce4-4b27-a077-68b3819d0c9a","message_receipt_time":"2023-07-19T07:05:14.799Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"invalid.json","elapsed_time_ns":23443042,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.823+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":351},"message":"End processing SQS S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"5fafe7f0-4ce4-4b27-a077-68b3819d0c9a","message_receipt_time":"2023-07-19T07:05:14.799Z","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-07-19T09:05:14.823+0200","log.logger":"aws-s3.sqs","log.origin":{"file.name":"awss3/sqs.go","file.line":96},"message":"Failed processing SQS message.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","error":{"message":"failed processing SQS message (it will return to queue after visibility timeout): failed processing S3 event for object key \"invalid.json\" in bucket \"filebeat-s3-integtest-localstack-9uoyjj\" (object record 1 of 1 in SQS notification): failed reading s3 object (elapsed_time_ns=23439333): failed to decode json: invalid character 'j' looking for beginning of value"},"message_id":"5fafe7f0-4ce4-4b27-a077-68b3819d0c9a","elapsed_time_ns":23797583,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.839+0200","log.logger":"aws-s3.sqs","log.origin":{"file.name":"awss3/sqs.go","file.line":80},"message":"Received 1 SQS messages.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.839+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":303},"message":"SQS message contained 1 S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"8af9e7c2-e5ab-44c1-9ae7-9930900c3890","message_receipt_time":"2023-07-19T07:05:14.839Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.839+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":124},"message":"Begin S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"8af9e7c2-e5ab-44c1-9ae7-9930900c3890","message_receipt_time":"2023-07-19T07:05:14.839Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"multiline.json.gz","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.855+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":132},"message":"End S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"8af9e7c2-e5ab-44c1-9ae7-9930900c3890","message_receipt_time":"2023-07-19T07:05:14.839Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"multiline.json.gz","elapsed_time_ns":15461875,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.855+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":348},"message":"End processing SQS S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"8af9e7c2-e5ab-44c1-9ae7-9930900c3890","message_receipt_time":"2023-07-19T07:05:14.839Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.861+0200","log.logger":"aws-s3.sqs","log.origin":{"file.name":"awss3/sqs.go","file.line":80},"message":"Received 2 SQS messages.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.861+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":303},"message":"SQS message contained 1 S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"57dad83d-957c-4414-9cc3-8fcf2740f771","message_receipt_time":"2023-07-19T07:05:14.861Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.861+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":124},"message":"Begin S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"57dad83d-957c-4414-9cc3-8fcf2740f771","message_receipt_time":"2023-07-19T07:05:14.861Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"multiline.txt","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.861+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":303},"message":"SQS message contained 1 S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"96290c32-6fde-490b-be67-3e2ff1e3e1a4","message_receipt_time":"2023-07-19T07:05:14.861Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.861+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":124},"message":"Begin S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"96290c32-6fde-490b-be67-3e2ff1e3e1a4","message_receipt_time":"2023-07-19T07:05:14.861Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"log.txt","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.881+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":132},"message":"End S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"57dad83d-957c-4414-9cc3-8fcf2740f771","message_receipt_time":"2023-07-19T07:05:14.861Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"multiline.txt","elapsed_time_ns":19925208,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.881+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":348},"message":"End processing SQS S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"57dad83d-957c-4414-9cc3-8fcf2740f771","message_receipt_time":"2023-07-19T07:05:14.861Z","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.883+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/s3_objects.go","file.line":132},"message":"End S3 object processing.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"96290c32-6fde-490b-be67-3e2ff1e3e1a4","message_receipt_time":"2023-07-19T07:05:14.861Z","bucket_arn":"filebeat-s3-integtest-localstack-9uoyjj","object_key":"log.txt","elapsed_time_ns":21265583,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-19T09:05:14.883+0200","log.logger":"aws-s3.sqs_s3_event","log.origin":{"file.name":"awss3/sqs_s3_event.go","file.line":348},"message":"End processing SQS S3 event notifications.","id":"test_id","queue_url":"http://localhost:4566/000000000000/filebeat-sqs-integtest-localstack-9uoyjj","message_id":"96290c32-6fde-490b-be67-3e2ff1e3e1a4","message_receipt_time":"2023-07-19T07:05:14.861Z","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-19T09:05:29.782+0200","log.logger":"metric_registry","log.origin":{"file.name":"inputmon/input.go","file.line":70},"message":"unregistering","input_type":"aws-s3","id":"test_id","key":"test_id","uuid":"4a441e83-60d9-4a70-85be-b1f6dc1fba04","ecs.version":"1.6.0"}
--- PASS: TestInputRunSQSOnLocalstack (15.19s)
PASS
ok  	github.com/elastic/beats/v7/x-pack/filebeat/input/awss3	15.724s

mergify · 2023-06-09T09:04:31Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @bhapas? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2023-06-09T10:06:26Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-26T17:02:58.358+0000
Duration: 179 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	30643
Skipped	2208
Total	32851

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-07-19T12:31:17Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-07-19T12:31:19Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

bhapas · 2023-07-19T12:50:14Z

/test

rdner

@pierrehilbert since you tried out Terraform recently, you might want to have a look at this.

LGTM, only minor formatting suggestions from me.

x-pack/filebeat/input/awss3/_meta/terraform/README.md

x-pack/filebeat/input/awss3/input_integration_test.go

bhapas · 2023-07-24T09:52:18Z

Depends on #36101

mergify · 2023-07-24T20:44:50Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b test-queue-localstack upstream/test-queue-localstack
git merge upstream/main
git push upstream test-queue-localstack

endorama

A minor nitpick for some indentation but looks good to me. Thanks for the contribution!

Jenkinsfile

…n localstack (elastic#35727) * Test integration tests on localstack * Add localstack container in CI * fix panic in SQS input metrics * Add documentation * Fix PR comments

Test integration tests on localstack

a9a3ea1

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 9, 2023

mergify bot assigned bhapas Jun 9, 2023

Merge branch 'main' into test-queue-localstack

5a81733

bhapas force-pushed the test-queue-localstack branch 2 times, most recently from 4536076 to 887bdf0 Compare July 10, 2023 12:21

bhapas changed the title ~~Test integration tests on localstack~~ [awss3][integration test]Test integration tests on localstack Jul 10, 2023

bhapas changed the title ~~[awss3][integration test]Test integration tests on localstack~~ x-pack/filebeat/input/awss3[integration test]Test integration tests on localstack Jul 10, 2023

bhapas force-pushed the test-queue-localstack branch from 887bdf0 to 4eb901c Compare July 10, 2023 12:25

Fix localstack integration test

0adb357

bhapas force-pushed the test-queue-localstack branch from 4eb901c to 0adb357 Compare July 10, 2023 12:31

bhapas added the Team:Security-External Integrations label Jul 10, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 10, 2023

bhapas added backport-skip Skip notification from the automated backport with mergify 8.10-candidate labels Jul 10, 2023

Merge branch 'main' into test-queue-localstack

b4d49d4

bhapas force-pushed the test-queue-localstack branch 2 times, most recently from baf3198 to 4d81d68 Compare July 10, 2023 14:53

Add localstack container in CI

58d76de

bhapas force-pushed the test-queue-localstack branch 7 times, most recently from 550ca31 to d17c60b Compare July 11, 2023 13:42

Merge branch 'main' into test-queue-localstack

601ae90

bhapas force-pushed the test-queue-localstack branch from d17c60b to 601ae90 Compare July 12, 2023 09:58

bhapas and others added 2 commits July 19, 2023 09:06

Merge branch 'main' into test-queue-localstack

f32cc3b

Refactor files to fix some issues

c344636

bhapas added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:Cloud-Monitoring Label for the Cloud Monitoring team >enhancement Filebeat Filebeat labels Jul 19, 2023

bhapas marked this pull request as ready for review July 19, 2023 12:31

bhapas requested review from a team as code owners July 19, 2023 12:31

bhapas requested review from belimawr and faec July 19, 2023 12:31

bhapas added 2 commits July 20, 2023 09:32

Add documentation

4fb2fdb

Merge remote-tracking branch 'upstream/main' into test-queue-localstack

fe43bb8

botelastic bot added the Team:Automation Label for the Observability productivity team label Jul 20, 2023

rdner reviewed Jul 20, 2023

View reviewed changes

x-pack/filebeat/input/awss3/_meta/terraform/README.md Show resolved Hide resolved

x-pack/filebeat/input/awss3/input_integration_test.go Outdated Show resolved Hide resolved

rdner approved these changes Jul 20, 2023

View reviewed changes

bhapas added 2 commits July 25, 2023 06:19

Merge remote-tracking branch 'upstream/main' into test-queue-localstack

e46df6c

Fix PR comments

207e80d

bhapas force-pushed the test-queue-localstack branch from 26dff85 to 207e80d Compare July 25, 2023 07:39

Merge branch 'main' into test-queue-localstack

d190e7d

endorama approved these changes Jul 27, 2023

View reviewed changes

Jenkinsfile Show resolved Hide resolved

Jenkinsfile Show resolved Hide resolved

bhapas merged commit 6a5b09a into elastic:main Jul 27, 2023

bhapas deleted the test-queue-localstack branch July 27, 2023 09:33

bhapas mentioned this pull request May 22, 2024

[aws/s3]: fix s3_input integration test #39648

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/filebeat/input/awss3[integration test]Test integration tests on localstack #35727

x-pack/filebeat/input/awss3[integration test]Test integration tests on localstack #35727

bhapas commented Jun 9, 2023 •

edited

Loading

mergify bot commented Jun 9, 2023

elasticmachine commented Jun 9, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 19, 2023

elasticmachine commented Jul 19, 2023

bhapas commented Jul 19, 2023

rdner left a comment

bhapas commented Jul 24, 2023

mergify bot commented Jul 24, 2023

endorama left a comment

x-pack/filebeat/input/awss3[integration test]Test integration tests on localstack #35727

x-pack/filebeat/input/awss3[integration test]Test integration tests on localstack #35727

Conversation

bhapas commented Jun 9, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Logs

mergify bot commented Jun 9, 2023

elasticmachine commented Jun 9, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Jul 19, 2023

elasticmachine commented Jul 19, 2023

bhapas commented Jul 19, 2023

rdner left a comment

Choose a reason for hiding this comment

bhapas commented Jul 24, 2023

mergify bot commented Jul 24, 2023

endorama left a comment

Choose a reason for hiding this comment

bhapas commented Jun 9, 2023 •

edited

Loading

elasticmachine commented Jun 9, 2023 •

edited by jenkins-beats-ci bot

Loading