[Filebeat] Investigate adding cloudwatch input

[kaiyan-sheng]

For collecting CloudWatch logs, CloudWatch API GetLogEvents or FilterLogEvents can be used. GetLogEvents can be used to lists log events from a specified log stream. FilterLogEvents can be used to list log events from the specified log group.

Limitation 1:
Using these two CloudWatch API to query logs is not very scalable due to the transactions per second (TPS) limits for GetLogEvents and FilterLogEvents:

• GetLogEvents API quota: 10 requests per second per account per region
• FilterLogEvents API: 5 transactions per second/account/region
• DescribeLogGroups API: 5 transactions per second/account/region
• DescribeLogStreams API: 5 transactions per second/account/region

Limitation 2:
By default, GetLogEvents and FilterLogEvents operations return as many log events as can fit in 1 MB (up to 10,000 log events), or all the events found within the time range that you specify. If the results include a token, then there are more log events available, and you can get additional results by specifying the token in a subsequent call.

Advantage:
Using CloudWatch API to get logs is much cheaper than using lambda functions.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

Just for the record, here is the original draft PR: #16524

[kaiyan-sheng]

After comparing FilterLogEvents API and GetLogEvents API, we decide to use FilterLogEvents in awscloudwatch Filebeat input.

Several main reasons:

1. FilterLogEvents API makes requests per log group and GetLogEvents API makes requests per log stream. When there are multiple log streams under the same log group, using FilterLogEvents means less API call.
2. GetLogEvents API requires both log group name and log stream name to be given. This means either user has to list all log stream names or Filebeat has to make an extra API call to list all log stream names under a given log group.
3. FilterLogEvents API has a filterPattern input parameter can be used to filter only collecting logs you are interested in. (This is not in the first version of aws cloudwatch input.)