filter_ecs: new filter for AWS ECS Metadata (#5898)

* filter_ecs: new filter for AWS ECS Metadata The filter is built primarily for a daemon deployment mode where Fluent Bit runs once per node/instance and collects all log files on that host. The filter can attach metadata to these logs. But- what if there is a container running on the host that is not part of an ECS Task? Json-file log driver files are written to disk with only the container ID to distinguish where they came from. So the daemon would still collect logs from containers not part of a task and there is no easy way to ignore those logs. Without this patch, in that case, the Fluent Bit output would be spammed with constant errors from this filter. This patch suppresses failures after 2 failed attempts. Signed-off-by: Wesley Pettit <[email protected]>
fluent · Nov 18, 2022 · 0f46cf3 · 0f46cf3
1 parent 300206a
commit 0f46cf3
Show file tree

Hide file tree

Showing 7 changed files with 2,238 additions and 0 deletions.
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -206,6 +206,7 @@ option(FLB_OUT_WEBSOCKET      "Enable Websocket output plugin"      Yes)
 option(FLB_FILTER_ALTER_SIZE  "Enable alter_size filter"            Yes)
 option(FLB_FILTER_AWS         "Enable aws filter"                   Yes)
 option(FLB_FILTER_CHECKLIST   "Enable checklist filter"             Yes)
+option(FLB_FILTER_ECS         "Enable AWS ECS filter"               Yes)
 option(FLB_FILTER_EXPECT      "Enable expect filter"                Yes)
 option(FLB_FILTER_GREP        "Enable grep filter"                  Yes)
 option(FLB_FILTER_MODIFY      "Enable modify filter"                Yes)

diff --git a/plugins/CMakeLists.txt b/plugins/CMakeLists.txt
@@ -272,6 +272,7 @@ REGISTER_OUT_PLUGIN("out_s3")
 REGISTER_FILTER_PLUGIN("filter_alter_size")
 REGISTER_FILTER_PLUGIN("filter_aws")
 REGISTER_FILTER_PLUGIN("filter_checklist")
+REGISTER_FILTER_PLUGIN("filter_ecs")
 REGISTER_FILTER_PLUGIN("filter_record_modifier")
 REGISTER_FILTER_PLUGIN("filter_throttle")
 REGISTER_FILTER_PLUGIN("filter_throttle_size")

diff --git a/plugins/filter_ecs/CMakeLists.txt b/plugins/filter_ecs/CMakeLists.txt
@@ -0,0 +1,5 @@
+set(src
+  ecs.c
+  )
+
+FLB_PLUGIN(filter_ecs "${src}" "")