out_cloudwatch: add account ID support for CloudWatch entity

[zhihonl]

Summary

CloudWatch entity needs account ID in the entity field to ensure entity does not leak outside the existing account. We also need to drop entity on client side if account ID is not filled out because backend does not have the functionality to smartly decide when to drop entity.

For example, if user A vends logs to user B, but account id is not in entity, backend will think the entity is sent to customer A's account, so user B will incorrectly get the entity from user A. In this case we need to drop the entity on client-side to prevent account information leaking outside the current account.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

Config without account ID set in aws plugin

[INPUT]
  Name                tail
  Tag                 application.*
  Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
  Path                /var/log/containers/*.log
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_container.db
  Mem_Buf_Limit       50MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Rotate_Wait         30
  storage.type        filesystem
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/fluent-bit*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_log.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/cloudwatch-agent*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_cwagent.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                aws
  Match               application.*
  enable_entity       true

[FILTER]
  Name                kubernetes
  Match               application.*
  Kube_URL            https://kubernetes.default.svc:443
  Kube_Tag_Prefix     application.var.log.containers.
  Merge_Log           On
  Merge_Log_Key       log_processed
  K8S-Logging.Parser  On
  K8S-Logging.Exclude Off
  Labels              Off
  Annotations         Off
  Use_Kubelet         On
  Kubelet_Port        10250
  Buffer_Size         0
  use_pod_association true
[OUTPUT]
  Name                cloudwatch_logs
  Match               application.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
  log_stream_prefix   ${HOST_NAME}-
  auto_create_group   true
  extra_user_agent    container-insights
  add_entity          true

• Debug log output from testing the change

Output without AccountId

We can see the entity is using the backend fallback entity

Output with AccountId when sending out of current account

Output without entity flags

This shows that we are still backward compatible

│ [2024/10/30 17:28:27] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] cloudwatch:PutLogEvents: events=1137, payload=1047258 bytes                                                                                                                                                                                               │
│ [2024/10/30 17:28:27] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream ip-192-168-63-18.ec2.internal-application.var.log.containers.fluent-bit-89wq8_amazon-cloudwatch_fluent-bit-84f776762608546d312d87eeb428c9c9f7bb164c35c97a754ad8797c43f474df.log                                          │
│ [2024/10/30 17:28:27] [debug] [upstream] KA connection #194 to logs.us-east-1.amazonaws.com:443 has been assigned (recycled)                                                                                                                                                                                                       │
│ [2024/10/30 17:28:27] [debug] [http_client] not using http_proxy for header                                                                                                                                                                                                                                                        │
│ [2024/10/30 17:28:27] [debug] [aws_credentials] Requesting credentials from the EC2 provider..                                                                                                                                                                                                                                     │
│ [2024/10/30 17:28:27] [debug] [upstream] KA connection #199 to logs.us-east-1.amazonaws.com:443 is now available                                                                                                                                                                                                                   │
│ [2024/10/30 17:28:27] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200                                                                                                                                                                                                                              │
│ [2024/10/30 17:28:27] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http data=HTTP/1.1 200 OK                                                                                                                                                                                                                    │
│ x-amzn-RequestId: fa396d26-8fae-4120-890e-bdeb7fbfb38b                                                                                                                                                                                                                                                                             │
│ Content-Type: application/x-amz-json-1.1                                                                                                                                                                                                                                                                                           │
│ Content-Length: 80                                                                                                                                                                                                                                                                                                                 │
│ Date: Wed, 30 Oct 2024 17:28:27 GMT                                                                                                                                                                                                                                                                                                │
│                                                                                                                                                                                                                                                                                                                                    │
│ {"nextSequenceToken":"49654323146657947237151620017781351458997669782897034498"}                                                                                                                                                                                                                                                   │
│ [2024/10/30 17:28:27] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http payload={"nextSequenceToken":"49654323146657947237151620017781351458997669782897034498"}                                                                                                                                                │
│ [2024/10/30 17:28:27] [debug] [upstream] KA connection #196 to logs.us-east-1.amazonaws.com:443 is now available                                                                                                                                                                                                                   │
│ [2024/10/30 17:28:27] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200                                                                                                                                                                                                                              │
│ [2024/10/30 17:28:27] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http data=HTTP/1.1 200 OK                                                                                                                                                                                                                    │
│ x-amzn-RequestId: 2910c0a8-4bcc-4110-903a-59cc8c88b761                                                                                                                                                                                                                                                                             │
│ Content-Type: application/x-amz-json-1.1                                                                                                                                                                                                                                                                                           │
│ Content-Length: 80                                                                                                                                                                                                                                                                                                                 │
│ Date: Wed, 30 Oct 2024 17:28:27 GMT                                                                                                                                                                                                                                                                                                │
│                                                                                                                                                                                                                                                                                                                                    │
│ {"nextSequenceToken":"49656145141491667532279081537089852841793243498340681506"}                                                                                                                                                                                                                                                   │
│ [2024/10/30 17:28:27] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http payload={"nextSequenceToken":"49656145141491667532279081537089852841793243498340681506"}

• Attached Valgrind output that shows no leaks or memory corruption was found

[ OK ]
==20691==
==20691== HEAP SUMMARY:
==20691==     in use at exit: 144 bytes in 1 blocks
==20691==   total heap usage: 6,761 allocs, 6,760 frees, 6,400,590 bytes allocated
==20691==
==20691== 144 bytes in 1 blocks are still reachable in loss record 1 of 1
==20691==    at 0x4C31C1D: calloc (vg_replace_malloc.c:1328)
==20691==    by 0x447E02: main (acutest.h:1632)
==20691==
==20691== LEAK SUMMARY:
==20691==    definitely lost: 0 bytes in 0 blocks
==20691==    indirectly lost: 0 bytes in 0 blocks
==20691==      possibly lost: 0 bytes in 0 blocks
==20691==    still reachable: 144 bytes in 1 blocks
==20691==         suppressed: 0 bytes in 0 blocks
==20691==
==20691== For lists of detected and suppressed errors, rerun with: -s
==20691== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
SUCCESS: All unit tests have passed.
==19745==
==19745== HEAP SUMMARY:
==19745==     in use at exit: 0 bytes in 0 blocks
==19745==   total heap usage: 2 allocs, 2 frees, 1,168 bytes allocated
==19745==
==19745== All heap blocks were freed -- no leaks are possible
==19745==
==19745== For lists of detected and suppressed errors, rerun with: -s
==19745== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[swapneils]

Can we also add a description in the PR overview as to what specific issue this is resolving?

[swapneils]

We need to discuss the implications of this issue and the proposed fix in more detail before merging this change, blocking the PR for now.

[swapneils]

Code concerns have been addressed, and post this change we don't expect any security risks from the Compass feature.

Approved; will wait on squash+merge until the time comes to deploy aws-for-fluent-bit with this change.