Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adds entity to PLE calls in cloudwatch logs plugin when used in EKS with kubernetes filter #2

Merged
merged 83 commits into from
Oct 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
efed0b6
Revert "out_datadog: fix/add error handling for all flb_sds calls (#5…
matthewfala Jan 7, 2023
71143a8
aws: add excluded_headers option to signv4 function
matthewfala Dec 20, 2022
3de95b3
out_es: aoss authentication support
matthewfala Dec 20, 2022
947b13e
out_opensearch: aoss authentication support
matthewfala Dec 20, 2022
6a9be1b
flb_aws_util: added function flb_aws_strftime_precision for time output.
Claych Nov 28, 2022
6e8e335
out_kinesis_streams: used function flb_aws_strftime_precision for tim…
Claych Dec 28, 2022
fefa130
out_kinesis_firehose: used function flb_aws_strftime_precision for ti…
Claych Dec 28, 2022
2ceb7ad
out_s3: fix logic in log_key warn message
PettitWesley Dec 14, 2022
88f4952
filter_ecs: all fixes from December 2022 in these PRs:
PettitWesley Jan 4, 2023
c2b00bd
edr-chunk-length-scraper
matthewfala Jan 7, 2023
7f76a49
edr-buffer-resize-logic
matthewfala Jan 7, 2023
ba4b5a2
datadog: resolve tag buffer resize bug
matthewfala Dec 16, 2022
8510a94
core: sync scheduler resolve multi task user issue
matthewfala Feb 9, 2023
cf65f13
aws: util: extra_user_agent is always type flb_sds_t
PettitWesley Dec 19, 2022
65c4e32
out_cloudwatch_logs: aws client extra_user_agent is always flb_sds_t
PettitWesley Dec 19, 2022
d872675
include: aws: remove unneeeded free_user_agent
PettitWesley Dec 20, 2022
2c763cf
upstream_conn: clean up keepalive event in sync case
PettitWesley Feb 13, 2023
52658cb
upstream_conn: fix ordering of mk_event_inject and prepare_conn_destroy
PettitWesley Feb 13, 2023
b9d990f
mk_event: fix bucket queue corruption in mk_event_add and init events…
PettitWesley Feb 14, 2023
88f2778
input_chunk: add context for chunk append debug message
PettitWesley Jan 31, 2023
29996cb
in_tail: add file name context to inotify debug event mask
PettitWesley Jan 31, 2023
cf37e73
in_tail: fix ordering of args in tail debug message
PettitWesley Apr 18, 2023
3ceaeff
out_s3: fix s3 key tag bug by using sds string
PettitWesley Mar 9, 2023
af46799
input_chunk: info level instead of debug for chunk removal msg (#6719)
PettitWesley Jan 28, 2023
33e5c86
input_chunk: use total_chunks_up in overlimit warn message (#6714)
PettitWesley Jan 28, 2023
6343e98
out_cloudwatch_logs: support tls verify and port options
matthewfala Mar 15, 2023
c8ea7f2
go plugins: separate exit and deregister into separate functions
PettitWesley Apr 4, 2023
1c4c8a0
proxy: go: fix build by removing return value for void proxy_go_destroy
PettitWesley Apr 13, 2023
eb4ccc8
sds: fix off by 1 bug in flb_sds_printf
PettitWesley Apr 7, 2023
1d7c0f6
sds: flb_sds_printf: flb_sds_increase increases by not to
PettitWesley Apr 10, 2023
2d939a0
tests: internal: sds: add test for #7143 off by 1 bug
PettitWesley Apr 10, 2023
7e07015
out_cloudwatch_logs: fix logic in free-ing log streams on shutdown
PettitWesley Apr 11, 2023
fac3e8f
time: fixed printf format string in flb_time_pop_from_mpack
leonardo-albertovich Apr 25, 2023
341febf
aws: sts_credentials: make STS response parsing less brittle by check…
PettitWesley May 3, 2023
9c3cfe1
config_map: fix possible NULL-deref
leonardo-albertovich Feb 17, 2023
4704ce9
engine: added retry cleanup when a task cannot be retried
leonardo-albertovich Feb 14, 2023
e0e6bcc
out_s3: fix incorrect decrementing of after UploadPart failure
PettitWesley May 16, 2023
f227ec9
lib: monkey: upgrade to 13a4ccd3 for AWS Distro
PettitWesley Jun 1, 2023
9112100
lib: chunkio: upgrade to v1.4.0
edsiper Dec 20, 2022
7b35f05
aws distro lib upgrades: libbacktrace to 8602fda, cfl to v0.2.3, onig…
PettitWesley Jun 1, 2023
be7b387
aws_util: memory fixes
PettitWesley Mar 9, 2023
6cdfe26
in_exec: fix printf-like format and arguments
ChezBunch Mar 3, 2023
85edd27
in_exec: use flb_calloc to fill NULL (#5715)
nokute78 Jul 14, 2022
3b9400a
in_tail: reset stat struct after usage
PettitWesley May 31, 2023
70be86c
in_http: fix parser memory reinitialization
tszshingt May 17, 2023
adc2b27
record_accessor: allow single character input in flb_ra_translate
nokute78 May 6, 2023
e4512fe
output_thread: added event memory intialization (#7303)
leonardo-albertovich May 3, 2023
6adf10a
filter_modify: fix clean up process
nokute78 May 14, 2023
db58092
aws: always use calloc for creds
PettitWesley Jun 3, 2023
622a81b
aws: check for full xml end node in flb_aws_xml_get_val
PettitWesley Jun 3, 2023
fec565a
aws: protect credential providers with pthread_mutex
PettitWesley Jun 3, 2023
eda6dd4
signv4: always use calloc
PettitWesley Jun 3, 2023
68405c0
cloudwatch_logs: remove sequence tokens from API calls
matthewfala Jan 26, 2023
2debef8
out_prometheus_remote_write: add authorization to prometheus remote w…
Claych Mar 9, 2023
5bad538
multiline: remove incorrect flush
May 24, 2023
68f8c05
tests: ml: update expectation based on updated flush logic
May 25, 2023
99bc266
upstream: decoupled socket shutdown from upstream connection release
leonardo-albertovich Jul 20, 2023
a346141
connection: a shutdown guard flag was added
leonardo-albertovich Jul 20, 2023
2dcffde
filter_throttle: fix print_status false log reduction option
matthewfala Oct 2, 2023
e2f3f18
out_cloudwatch_logs: add PutLogEvents entity scraping logic for names…
zhihonl Aug 30, 2024
8b949f5
out_cloudwatch_logs: add entity scraping logic for cluster and instan…
zhihonl Sep 5, 2024
52b4cf7
added logic to get pod to service map from endpoint and store it in s…
nathalapooja Sep 6, 2024
c5df964
filter_kubernetes: Fix segmentation fault for kubernetes unit tests (…
zhihonl Sep 17, 2024
af4f7da
filter_kubernetes: add pod to service map parsing logics (#13)
zhihonl Sep 19, 2024
00904de
filter_kubernetes: fix memory leaks in pod association components (#15)
zhihonl Sep 20, 2024
e0625ce
Create options_use-pod-association-enabled_fluent-bit.log
zhihonl Sep 20, 2024
2b400a7
filter_kubernetes: add service name source parsing logic to kubernete…
zhihonl Sep 25, 2024
c11c024
filter_kubernetes: add workload scraping logics for kubernetes filter…
zhihonl Sep 26, 2024
85866b1
Implement HTTPS pod to service association endpoint calls with mtls (…
nathalapooja Sep 27, 2024
247a36f
Add workload fallback logics for entity service name (#18)
zhihonl Oct 1, 2024
af03117
Fix fluentbit failure when running without permissions (#19)
zhihonl Oct 3, 2024
9875b8c
Add environment fallback for different kubernetes platforms (#20)
zhihonl Oct 11, 2024
8e6d48c
Fix incorrect kubernetes platform and missing entity in native Kubern…
zhihonl Oct 15, 2024
ca0749b
Fix to not send entity when flag is disabled (#23)
nathalapooja Oct 16, 2024
fac95cb
adds entity to PLE calls in cloudwatch logs when used in EKS with kub…
nathalapooja Oct 16, 2024
dc025f8
Remove entity related log messages and rename entity fields (#22)
zhihonl Oct 17, 2024
740a57a
Merge branch 'aws-fluent-bit-cherry-pick' of github.com:zhihonl/priva…
nathalapooja Oct 17, 2024
526512e
Remove entity related fields on top-level message pack map (#24)
zhihonl Oct 18, 2024
5dc2e63
Merge branch 'aws-fluent-bit-cherry-pick' of github.com:zhihonl/priva…
nathalapooja Oct 21, 2024
f46721f
Increment filter count when dynamic entity fields are modified (#25)
zhihonl Oct 22, 2024
ba00160
Move messagepack destroy to later stage to prevent memory issue (#26)
zhihonl Oct 22, 2024
a24118a
House cleaning changes to address upstream merge blockers (#27)
zhihonl Oct 22, 2024
9de7ed7
Merge branch 'aws-fluent-bit-cherry-pick' of github.com:zhihonl/priva…
nathalapooja Oct 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@
*~
_book/
lib/jemalloc
cmake-build-debug/
tests/internal/flb_tests_internal.h
tests/runtime/flb_tests_runtime.h
tests/internal/cmake-build-debug/
tests/runtime/cmake-build-debug/
build/*
include/fluent-bit/flb_info.h
include/fluent-bit/flb_plugins.h
Expand Down
3 changes: 3 additions & 0 deletions include/fluent-bit/flb_hash.h
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ struct flb_hash {
int max_entries;
int total_count;
int cache_ttl;
int force_remove_pointer;
size_t size;
struct mk_list entries;
struct flb_hash_table *table;
Expand All @@ -63,6 +64,8 @@ struct flb_hash {
struct flb_hash *flb_hash_create(int evict_mode, size_t size, int max_entries);
struct flb_hash *flb_hash_create_with_ttl(int cache_ttl, int evict_mode,
size_t size, int max_entries);
struct flb_hash *flb_hash_create_with_ttl_force_destroy(int cache_ttl, int evict_mode,
size_t size, int max_entries);
void flb_hash_destroy(struct flb_hash *ht);

int flb_hash_add(struct flb_hash *ht,
Expand Down
16 changes: 15 additions & 1 deletion plugins/filter_aws/aws.c
Original file line number Diff line number Diff line change
Expand Up @@ -558,14 +558,22 @@ static int cb_aws_filter(const void *data, size_t bytes,
ctx->availability_zone_len);
}

if (ctx->instance_id_include) {
if (ctx->instance_id_include && !ctx->enable_entity) {
msgpack_pack_str(&tmp_pck, FLB_FILTER_AWS_INSTANCE_ID_KEY_LEN);
msgpack_pack_str_body(&tmp_pck,
FLB_FILTER_AWS_INSTANCE_ID_KEY,
FLB_FILTER_AWS_INSTANCE_ID_KEY_LEN);
msgpack_pack_str(&tmp_pck, ctx->instance_id_len);
msgpack_pack_str_body(&tmp_pck,
ctx->instance_id, ctx->instance_id_len);
} else if (ctx->instance_id_include && ctx->enable_entity) {
msgpack_pack_str(&tmp_pck, FLB_FILTER_AWS_ENTITY_INSTANCE_ID_KEY_LEN);
msgpack_pack_str_body(&tmp_pck,
FLB_FILTER_AWS_ENTITY_INSTANCE_ID_KEY,
FLB_FILTER_AWS_ENTITY_INSTANCE_ID_KEY_LEN);
msgpack_pack_str(&tmp_pck, ctx->instance_id_len);
msgpack_pack_str_body(&tmp_pck,
ctx->instance_id, ctx->instance_id_len);
}

if (ctx->instance_type_include) {
Expand Down Expand Up @@ -740,6 +748,12 @@ static struct flb_config_map config_map[] = {
0, FLB_TRUE, offsetof(struct flb_filter_aws, hostname_include),
"Enable EC2 instance hostname"
},
{
FLB_CONFIG_MAP_BOOL, "enable_entity", "false",
0, FLB_TRUE, offsetof(struct flb_filter_aws, enable_entity),
"Enable entity prefix for fields used for constructing entity."
"This currently only affects instance ID"
},
{0}
};

Expand Down
8 changes: 8 additions & 0 deletions plugins/filter_aws/aws.h
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@
#define FLB_FILTER_AWS_AVAILABILITY_ZONE_KEY_LEN 2
#define FLB_FILTER_AWS_INSTANCE_ID_KEY "ec2_instance_id"
#define FLB_FILTER_AWS_INSTANCE_ID_KEY_LEN 15
#define FLB_FILTER_AWS_ENTITY_INSTANCE_ID_KEY "aws_entity_ec2_instance_id"
#define FLB_FILTER_AWS_ENTITY_INSTANCE_ID_KEY_LEN 26
#define FLB_FILTER_AWS_INSTANCE_TYPE_KEY "ec2_instance_type"
#define FLB_FILTER_AWS_INSTANCE_TYPE_KEY_LEN 17
#define FLB_FILTER_AWS_PRIVATE_IP_KEY "private_ip"
Expand Down Expand Up @@ -111,6 +113,12 @@ struct flb_filter_aws {
size_t hostname_len;
int hostname_include;

/*
* Enable entity prefix appending. This appends
* 'aws_entity' to relevant keys
*/
int enable_entity;

/* number of new keys added by this plugin */
int new_keys;

Expand Down
37 changes: 37 additions & 0 deletions plugins/filter_kubernetes/kube_conf.c
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,12 @@ struct flb_kube *flb_kube_conf_create(struct flb_filter_instance *ins,
ctx->api_https = FLB_FALSE;
}

if (ctx->use_pod_association) {
ctx->kubernetes_api_host = flb_strdup(FLB_API_HOST);
ctx->kubernetes_api_port = FLB_API_PORT;
}


}
else if (!url) {
ctx->api_host = flb_strdup(FLB_API_HOST);
Expand Down Expand Up @@ -190,6 +196,12 @@ struct flb_kube *flb_kube_conf_create(struct flb_filter_instance *ins,
flb_plg_info(ctx->ins, "https=%i host=%s port=%i",
ctx->api_https, ctx->api_host, ctx->api_port);
}


ctx->pod_hash_table = flb_hash_create_with_ttl_force_destroy(ctx->pod_service_map_ttl,
FLB_HASH_EVICT_OLDER,
FLB_HASH_TABLE_SIZE,
FLB_HASH_TABLE_SIZE);
return ctx;
}

Expand All @@ -203,6 +215,10 @@ void flb_kube_conf_destroy(struct flb_kube *ctx)
flb_hash_destroy(ctx->hash_table);
}

if (ctx->pod_hash_table) {
flb_hash_destroy(ctx->pod_hash_table);
}

if (ctx->merge_log == FLB_TRUE) {
flb_free(ctx->unesc_buf);
}
Expand All @@ -211,6 +227,9 @@ void flb_kube_conf_destroy(struct flb_kube *ctx)
if (ctx->parser == NULL && ctx->regex) {
flb_regex_destroy(ctx->regex);
}
if (ctx->deploymentRegex) {
flb_regex_destroy(ctx->deploymentRegex);
}

flb_free(ctx->api_host);
flb_free(ctx->token);
Expand All @@ -222,6 +241,24 @@ void flb_kube_conf_destroy(struct flb_kube *ctx)
flb_upstream_destroy(ctx->upstream);
}

if(ctx->pod_association_tls) {
flb_tls_destroy(ctx->pod_association_tls);
}

if (ctx->pod_association_upstream) {
flb_upstream_destroy(ctx->pod_association_upstream);
}

if (ctx->kubernetes_upstream) {
flb_upstream_destroy(ctx->kubernetes_upstream);
}
if (ctx->kubernetes_api_host) {
flb_free(ctx->kubernetes_api_host);
}
if (ctx->platform) {
flb_free(ctx->platform);
}

#ifdef FLB_HAVE_TLS
if (ctx->tls) {
flb_tls_destroy(ctx->tls);
Expand Down
72 changes: 72 additions & 0 deletions plugins/filter_kubernetes/kube_conf.h
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,40 @@
#define FLB_KUBE_TAG_PREFIX "kube.var.log.containers."
#endif

/*
* Maximum attribute length for Entity's KeyAttributes
* values
* https://docs.aws.amazon.com/applicationsignals/latest/APIReference/API_Service.html#:~:text=Maximum%20length%20of%201024.
*/
#define KEY_ATTRIBUTES_MAX_LEN 1024
#define SERVICE_NAME_SOURCE_MAX_LEN 64

/*
* Configmap used for verifying whether if FluentBit is
* on EKS or native Kubernetes
*/
#define KUBE_SYSTEM_NAMESPACE "kube-system"
#define AWS_AUTH_CONFIG_MAP "aws-auth"

/*
* Possible platform values for Kubernetes plugin
*/
#define NATIVE_KUBERNETES_PLATFORM "k8s"
#define EKS_PLATFORM "eks"

struct kube_meta;

struct service_attributes {
char name[KEY_ATTRIBUTES_MAX_LEN];
int name_len;
char environment[KEY_ATTRIBUTES_MAX_LEN];
int environment_len;
char name_source[SERVICE_NAME_SOURCE_MAX_LEN];
int name_source_len;
int fields;

};

/* Filter context */
struct flb_kube {
/* Configuration parameters */
Expand Down Expand Up @@ -119,6 +151,7 @@ struct flb_kube {

/* Regex context to parse records */
struct flb_regex *regex;
struct flb_regex *deploymentRegex;
struct flb_parser *parser;

/* TLS CA certificate file */
Expand Down Expand Up @@ -158,6 +191,45 @@ struct flb_kube {

int kube_meta_cache_ttl;

/* Configuration used for enabling pod to service name mapping*/
int use_pod_association;
char *pod_association_host;
char *pod_association_endpoint;
int pod_association_port;

/*
* TTL is used to check how long should the mapped entry
* remain in the hash table
*/
struct flb_hash *pod_hash_table;
int pod_service_map_ttl;
int pod_service_map_refresh_interval;
flb_sds_t pod_service_preload_cache_path;
struct flb_upstream *pod_association_upstream;
/*
* This connection is used for calling Kubernetes configmaps
* endpoint so pod association can determine the environment.
* Example: EKS or Native Kubernetes.
*/
char *kubernetes_api_host;
int kubernetes_api_port;
struct flb_upstream *kubernetes_upstream;
char *platform;
/*
* This value is used for holding the platform config
* value. Platform will be overriden with this variable
* if it's set
*/
char *set_platform;

//Agent TLS certs
struct flb_tls *pod_association_tls;
char *pod_association_host_server_ca_file;
char *pod_association_host_client_cert_file;
char *pod_association_host_client_key_file;
int pod_association_host_tls_debug;
int pod_association_host_tls_verify;

struct flb_tls *tls;

struct flb_config *config;
Expand Down
Loading
Loading