AWS S3 notification receiver (SQS) improvement

[GGonzalezGomez]

Component(s)

receiver/awss3

Is your feature request related to a problem? Please describe.

No, it is extending the functionality of the component.

Describe the solution you'd like

Current functionality for the aws s3 receiver is reading a set of files from S3 based on the configuration. This allows processing of files archived in S3. The proposal (already mentioned in #30750) is to allow the receiver to read a SQS queue for S3 notifications. That way, when a new file is created in the bucket, a notification is sent to SQS. The receiver would be reading that queue waiting for new messages and when a notification is sent, read the message, retrieve the object key from it, and get the events from the file.

The heavy lifting of the component is already present as it reads files from S3. Component extension would be to add the possibility to configure the queue to listen to, read those messages for s3 notifications and get the file to read instead of just listing the objects in the bucket.

This would allow sending files to S3 for processing, not just once but for any file created in the bucket.

Describe alternatives you've considered

No response

Additional context

To give some background, this is for example already possible for logs in vector with this source https://vector.dev/docs/reference/configuration/sources/aws_s3

[github-actions]

Pinging code owners:

• receiver/awss3: @atoulme @adcharre

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[adcharre]

Thank you @GGonzalezGomez I had been meaning to raise this as a separate issue. I think this makes sense and there definitely appears to be a demand for it. I would expand this slightly to also include SNS as well.

[VihasMakwana]

Removing needs triage label based on codeowner's response.