Functionbeat module support

[ph]

I am starting to think how beatless modules will look like, how we can make it easy for users to get started with beatless. It looks a bit more complicated so I think its good to take the time to do it right.

At his core beatless is pretty similar to Filebeat except that it runs as a function on a serverless provider like AWS lambda and the execution model is synchronous instead of asynchronous. The data received could be either custom or an already existing log format that we might already know how to parse. I imagine that the same data can be received from 2 different trigger, cloudwatch or an SQS stream.

Inside beatless the following pieces are available to control:

• Provider: AWS, GCP or any other serverless infrastructure.
• Trigger: How the data is received, this could be Kinesis, SQS, Cloudfront.
• Processors: All the processor that Libbeat provide are available in Beatless.

Can we leverage what we already have in Filebeat?

If we look at Filebeat modules what could be used directly in beatless:

Available in FB	Can be used in Beatless without change?
path	NO
ingest pipeline	YES
ML Job	YES
Dashboards	YES
Fields	YES
Processors	YES
Tests	Maybe not directly

Looking at the above table only the path cannot be used directly as a value for beatless because its platform specific.

Questions or feedback?

• Is trying to offert FB modules in beatless a good idea?
• Is there any beatless specific module that FB will not have?
• Can we make the common parts of FB modules available outside of Filebeat?
• Where beat specific stuff will be defined (AKA Path for Filebeat)?
• How we will configure the source of the data it is done by the module definition or the users will have to configure it manualy. ie: use aws/cloudwatch with this logstream and this module will take care of the parsing.

Will like your feedback / ideas on this.

[ruflin]

I think we should make modules available in Beatless. There should be a module structure which works across Beats. Several discussion around a more general module structure happened in the past and this seems to be a good time now to tackle it.

One thing that happens with the above is that it decouples the way we gather the data (input/path) from the module. In Filebeat we will also need this decoupling of input from Modules as potentially multiple modules could receive the events through the same journald / syslog input.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlind23]

We endeavor to address Functionbeat use cases via the new Serverless Forwarder project (https://github.com/elastic/elastic-serverless-forwarder)with a better integration with serverless functionality provided by major cloud providers. Enhancements to the current Functionbeat will therefore be limited.