[MongoDB Atlas] Add support of mongodb database datastream #9539

milan-elastic · 2024-04-08T10:25:17Z

Create the package for Mongodb Atlas Integration and Provide the support of Mongod database logs.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

Along with the mongod_database PR, to keep the code changes(review comments) consistent with other log datastreams, I have made the same changes for mongod_audit datastream.

All changes

Change follows the contributing guidelines
Supported versions of the monitoring target are documented
Supported operating systems are documented (if applicable)
Integration or System tests exist
Documentation exists, useful guidelines to follow
Fields follow ECS and naming conventions
At least a manual test with ES / Kibana / Agent has been performed.
Required Kibana version set to: 8.12.0
TSDB enablement for the data stream

Data dataset changes

This entry is currently only recommended. It will be mandatory once we provide better support for it.

Sample event (sample_event.json) exists

Related Issue

Relates https://github.com/elastic/obs-infraobs-team/issues/1374

Screenshots

Known Issues:

Currently Following challenges are still open to solve for this PR:

packages/mongodb_atlas/_dev/build/docs/README.md

Co-authored-by: subham sarkar <[email protected]>

…c/integrations into mongodb-atlas-database-logs

muthu-mps

Please add different log samples in the pipeline test.

elasticmachine · 2024-04-15T10:01:37Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

milan-elastic · 2024-04-15T11:01:15Z

Please add different log samples in the pipeline test.

Done, added more events for pipeline that we are managed to generate!

SubhrataK · 2024-04-16T12:20:00Z

Please move the line graphs together.

milan-elastic · 2024-04-16T17:40:34Z

Please move the line graphs together.

Done!

packages/mongodb_atlas/_dev/build/docs/README.md

packages/mongodb_atlas/data_stream/mongod_database/agent/stream/input.yml.hbs

packages/mongodb_atlas/_dev/build/docs/README.md

shmsr · 2024-04-17T07:49:30Z

Review comment:

diff --git a/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml b/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
index 5b34df3d8..e6ad6f0f1 100644
--- a/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
@@ -46,25 +46,27 @@ processors:
             value: '{{ _ingest.on_failure_message }}'
   - script:
       lang: painless
-      description: Provides descriptive value to the field.
+      description: Maps log severity levels to descriptive values
       tag: informative_log_level
       on_failure:
         - append:
             field: error.message
             value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
+      params:
+        severity_levels:
+          F: fatal
+          E: error
+          W: warning
+          I: informational
+          D1: debug 1
+          D2: debug 2
+          D3: debug 3
+          D4: debug 4
+          D5: debug 5
       source: |
-        Map m = new HashMap();
-        m.put("F", "fatal");
-        m.put("E", "error");
-        m.put("W", "warning");
-        m.put("I", "informational");
-        m.put("D1", "debug 1");
-        m.put("D2", "debug 2");
-        m.put("D3", "debug 3");
-        m.put("D4", "debug 4");
-        m.put("D5", "debug 5");
-        if (m.get(ctx.json?.s) != null) {
-          ctx.severity = m.get(ctx.json.s);
+        String severity = ctx.json?.s;
+        if (severity != null) {
+          ctx.log.level = params.severity_levels.getOrDefault(severity, null);
         }
   - rename:
       field: severity
@@ -101,20 +103,27 @@ processors:
   - script:
       lang: painless
       source: |-
-        boolean drop(Object o) {
-          if (o == null || o == '') {
-            return true;
+        /**
+         * Recursively drops null and empty values from the Elasticsearch doc context.
+         *
+         * @param o The object to process (can be a Map, List, or any other type).
+         */
+        void drop(Object o) {
+          if (o instanceof List) {
+            ((List) o).removeIf(item -> {
+              drop(item);
+              return item == null;
+            });
           } else if (o instanceof Map) {
-            ((Map) o).values().removeIf(v -> drop(v));
-            return (((Map) o).size() == 0);
-          } else if (o instanceof List) {
-            ((List) o).removeIf(v -> drop(v));
-            return (((List) o).length == 0);
+            ((Map) o).values().removeIf(item -> {
+              drop(item);
+              return item == null;
+            });
           }
-          return false;
         }
+
         drop(ctx);
-      description: Drops null/empty values recursively.
+      description: Drops null and empty values recursively from the Elasticsearch document context.
   - remove:
       field:
         - event.original
@@ -125,6 +134,7 @@ processors:
       field:
         - json
       ignore_missing: true
+      description: Removes temporary fields.
   - set:
       field: event.kind
       value: pipeline_error
@@ -136,4 +146,4 @@ on_failure:
   - append:
       field: event.kind
       value: pipeline_error
-      allow_duplicates: false
+      allow_duplicates: false
\ No newline at end of file

Improvements:

Use params. Not Hashmaps. (See: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-using.html#prefer-params)
No return value is required by drop as there's no one using it. Changed it to void. Also, made some minor changes.

shmsr · 2024-04-17T08:17:14Z

I think there's scope to make this CEL program better. I might have a few suggestions.

shmsr · 2024-04-18T05:35:05Z

@milan-elastic Need some clarity on this.

While trying to improve the CEL code, I was reading MongoDB's doc as well. Here's what I saw:

API that we are using is deprecated. Did we see this? Support is until June of 2025. We should invest some time if there's a new API and if the new API is better. See this.
Also, for logName parameter in the API, there are four options. Why did you specifically go for mongodb and not mongodb-audit-log? See this.
Also, have we mentioned that auditing logs are not there in M0, M2, M5, or the Serverless instances?
Read this full. This too.
We can take relevant parts of this doc to improve our doc.

milan-elastic · 2024-04-18T11:34:02Z

While trying to improve the CEL code, I was reading MongoDB's doc as well. Here's what I saw:

API that we are using is deprecated. Did we see this? Support is until June of 2025. We should invest some time if there's a new API and if the new API is better. See this.

The API itself is not deprecated; only the resource version we specified in the header during API calls will be deprecated. You can find more information about this version schema and the header here. To accommodate these changes in the version header, we have implemented a dynamic header that is based on the current year. The code for these changes is specified below.

"Header": {
              "Accept": ["application/vnd.atlas." + string(now.getFullYear()) + "-01-01+json"]
            }

Also, for logName parameter in the API, there are four options. Why did you specifically go for mongodb and not mongodb-audit-log? See this.

Yes, that is correct! Currently, we are collecting two types of logs from MongoDB Atlas: mongod_database logs and mongod_audit logs. To collect mongod_database logs, we use the log name mongodb, and for mongod_audit logs, we use mongodb-audit-log file. So both the logs are getting collected but under different datastreams. More bifurcation is provided here.

Also, have we mentioned that auditing logs are not there in M0, M2, M5, or the Serverless instances?

Sure, I will add that in Readme.

Read this full. This too.

We can take relevant parts of this doc to improve our doc.

Yes we have already visited this document and captured the information that found useful.

milan-elastic · 2024-04-18T11:46:39Z

Review comment:

diff --git a/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml b/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
index 5b34df3d8..e6ad6f0f1 100644
--- a/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
@@ -46,25 +46,27 @@ processors:
             value: '{{ _ingest.on_failure_message }}'
   - script:
       lang: painless
-      description: Provides descriptive value to the field.
+      description: Maps log severity levels to descriptive values
       tag: informative_log_level
       on_failure:
         - append:
             field: error.message
             value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
+      params:
+        severity_levels:
+          F: fatal
+          E: error
+          W: warning
+          I: informational
+          D1: debug 1
+          D2: debug 2
+          D3: debug 3
+          D4: debug 4
+          D5: debug 5
       source: |
-        Map m = new HashMap();
-        m.put("F", "fatal");
-        m.put("E", "error");
-        m.put("W", "warning");
-        m.put("I", "informational");
-        m.put("D1", "debug 1");
-        m.put("D2", "debug 2");
-        m.put("D3", "debug 3");
-        m.put("D4", "debug 4");
-        m.put("D5", "debug 5");
-        if (m.get(ctx.json?.s) != null) {
-          ctx.severity = m.get(ctx.json.s);
+        String severity = ctx.json?.s;
+        if (severity != null) {
+          ctx.log.level = params.severity_levels.getOrDefault(severity, null);
         }
   - rename:
       field: severity
@@ -101,20 +103,27 @@ processors:
   - script:
       lang: painless
       source: |-
-        boolean drop(Object o) {
-          if (o == null || o == '') {
-            return true;
+        /**
+         * Recursively drops null and empty values from the Elasticsearch doc context.
+         *
+         * @param o The object to process (can be a Map, List, or any other type).
+         */
+        void drop(Object o) {
+          if (o instanceof List) {
+            ((List) o).removeIf(item -> {
+              drop(item);
+              return item == null;
+            });
           } else if (o instanceof Map) {
-            ((Map) o).values().removeIf(v -> drop(v));
-            return (((Map) o).size() == 0);
-          } else if (o instanceof List) {
-            ((List) o).removeIf(v -> drop(v));
-            return (((List) o).length == 0);
+            ((Map) o).values().removeIf(item -> {
+              drop(item);
+              return item == null;
+            });
           }
-          return false;
         }
+
         drop(ctx);
-      description: Drops null/empty values recursively.
+      description: Drops null and empty values recursively from the Elasticsearch document context.
   - remove:
       field:
         - event.original
@@ -125,6 +134,7 @@ processors:
       field:
         - json
       ignore_missing: true
+      description: Removes temporary fields.
   - set:
       field: event.kind
       value: pipeline_error
@@ -136,4 +146,4 @@ on_failure:
   - append:
       field: event.kind
       value: pipeline_error
-      allow_duplicates: false
+      allow_duplicates: false
\ No newline at end of file

Improvements:

Use params. Not Hashmaps. (See: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-using.html#prefer-params)
No return value is required by drop as there's no one using it. Changed it to void. Also, made some minor changes.

+        /**
+         * Recursively drops null and empty values from the Elasticsearch doc context.
+         *
+         * @param o The object to process (can be a Map, List, or any other type).
+         */
        void drop(Object o) {
         if (o instanceof List) {
            ((List) o).removeIf(item -> {
              drop(item);
              return item == null;
            });
             ((Map) o).values().removeIf(item -> {
              drop(item);
              return item == null;
            });
           }
         }

Above function will not work when we have field with empty value. Like field1 = “”. Then this function will not remove this field from the event. It’s only work if field have null value. While the current script is handling mentioned case as well.

shmsr · 2024-04-18T11:56:30Z

While trying to improve the CEL code, I was reading MongoDB's doc as well. Here's what I saw:

API that we are using is deprecated. Did we see this? Support is until June of 2025. We should invest some time if there's a new API and if the new API is better. See this.

The API itself is not deprecated; only the resource version we specified in the header during API calls will be deprecated. You can find more information about this version schema and the header here. To accommodate these changes in the version header, we have implemented a dynamic header that is based on the current year. The code for these changes is specified below.
"Header": {
              "Accept": ["application/vnd.atlas." + string(now.getFullYear()) + "-01-01+json"]
            }
Also, for logName parameter in the API, there are four options. Why did you specifically go for mongodb and not mongodb-audit-log? See this.

Yes, that is correct! Currently, we are collecting two types of logs from MongoDB Atlas: mongod_database logs and mongod_audit logs. To collect mongod_database logs, we use the log name mongodb, and for mongod_audit logs, we use mongodb-audit-log file. So both the logs are getting collected but under different datastreams. More bifurcation is provided here.

Also, have we mentioned that auditing logs are not there in M0, M2, M5, or the Serverless instances?

Sure, I will add that in Readme.

Read this full. This too.

We can take relevant parts of this doc to improve our doc.

Yes we have already visited this document and captured the information that found useful.

Thanks for clarifying the points ++

shmsr · 2024-04-18T12:00:41Z

Review comment:

diff --git a/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml b/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
index 5b34df3d8..e6ad6f0f1 100644
--- a/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml
@@ -46,25 +46,27 @@ processors:
             value: '{{ _ingest.on_failure_message }}'
   - script:
       lang: painless
-      description: Provides descriptive value to the field.
+      description: Maps log severity levels to descriptive values
       tag: informative_log_level
       on_failure:
         - append:
             field: error.message
             value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
+      params:
+        severity_levels:
+          F: fatal
+          E: error
+          W: warning
+          I: informational
+          D1: debug 1
+          D2: debug 2
+          D3: debug 3
+          D4: debug 4
+          D5: debug 5
       source: |
-        Map m = new HashMap();
-        m.put("F", "fatal");
-        m.put("E", "error");
-        m.put("W", "warning");
-        m.put("I", "informational");
-        m.put("D1", "debug 1");
-        m.put("D2", "debug 2");
-        m.put("D3", "debug 3");
-        m.put("D4", "debug 4");
-        m.put("D5", "debug 5");
-        if (m.get(ctx.json?.s) != null) {
-          ctx.severity = m.get(ctx.json.s);
+        String severity = ctx.json?.s;
+        if (severity != null) {
+          ctx.log.level = params.severity_levels.getOrDefault(severity, null);
         }
   - rename:
       field: severity
@@ -101,20 +103,27 @@ processors:
   - script:
       lang: painless
       source: |-
-        boolean drop(Object o) {
-          if (o == null || o == '') {
-            return true;
+        /**
+         * Recursively drops null and empty values from the Elasticsearch doc context.
+         *
+         * @param o The object to process (can be a Map, List, or any other type).
+         */
+        void drop(Object o) {
+          if (o instanceof List) {
+            ((List) o).removeIf(item -> {
+              drop(item);
+              return item == null;
+            });
           } else if (o instanceof Map) {
-            ((Map) o).values().removeIf(v -> drop(v));
-            return (((Map) o).size() == 0);
-          } else if (o instanceof List) {
-            ((List) o).removeIf(v -> drop(v));
-            return (((List) o).length == 0);
+            ((Map) o).values().removeIf(item -> {
+              drop(item);
+              return item == null;
+            });
           }
-          return false;
         }
+
         drop(ctx);
-      description: Drops null/empty values recursively.
+      description: Drops null and empty values recursively from the Elasticsearch document context.
   - remove:
       field:
         - event.original
@@ -125,6 +134,7 @@ processors:
       field:
         - json
       ignore_missing: true
+      description: Removes temporary fields.
   - set:
       field: event.kind
       value: pipeline_error
@@ -136,4 +146,4 @@ on_failure:
   - append:
       field: event.kind
       value: pipeline_error
-      allow_duplicates: false
+      allow_duplicates: false
\ No newline at end of file

Improvements:

Use params. Not Hashmaps. (See: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-using.html#prefer-params)
No return value is required by drop as there's no one using it. Changed it to void. Also, made some minor changes.

+        /**
+         * Recursively drops null and empty values from the Elasticsearch doc context.
+         *
+         * @param o The object to process (can be a Map, List, or any other type).
+         */
        void drop(Object o) {
         if (o instanceof List) {
            ((List) o).removeIf(item -> {
              drop(item);
              return item == null;
            });
             ((Map) o).values().removeIf(item -> {
              drop(item);
              return item == null;
            });
           }
         }

Above function will not work when we have field with empty value. Like field1 = “”. Then this function will not remove this field from the event. It’s only work if field have null value. While the current script is handling mentioned case as well.

Okay, can we modify that program to add that condition? Also, this should also work:

void drop(Map map) {
  for (String key : map.keySet()) {
    if (map[key] instanceof Map) {
      drop(map[key]);
    } else if (map[key] == null || map[key].equals("")) {
      map.remove(key);
    }
  }
}

drop(ctx);

I just want to keep it simple and avoid returning a bool.

milan-elastic · 2024-04-18T12:31:27Z

void drop(Map map) {
for (String key : map.keySet()) {
if (map[key] instanceof Map) {
drop(map[key]);
} else if (map[key] == null || map[key].equals("")) {
map.remove(key);
}
}
}

drop(ctx);

It seems like this script might not handle few cases, I need to test it thoroughly. Current script is working fine without any issue as of now so I'm keeping it as it is for the time being. But once I test it thoroughly, I'll update this script in other datastreams as well in separate PR.

harnish-elastic

LGTM! 🚀

niraj-elastic

LGTM!

packages/mongodb_atlas/data_stream/mongod_database/fields/fields.yml

packages/mongodb_atlas/data_stream/mongod_database/manifest.yml

ishleenk17

Please recheck if there are any nit comments left.
Else Looks good!

elastic-sonarqube · 2024-04-18T16:36:14Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
93.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-04-18T16:37:10Z

💚 Build Succeeded

Buildkite Build
Commit: d3e36f4

History

💚 Build #10580 succeeded e760390
💚 Build #10492 succeeded 087e938
💔 Build #10489 failed 32b34ca
💚 Build #10486 succeeded 297634e
💚 Build #10429 succeeded 76ee648
💚 Build #10424 succeeded 3e248e5

elasticmachine · 2024-04-18T16:51:09Z

Package mongodb_atlas - 0.0.3 containing this change is available at https://epr.elastic.co/search?package=mongodb_atlas

milan-elastic added 2 commits April 8, 2024 15:45

add support of mongodb database datastream

2209b62

update changelog with pr link

0f8613e

milan-elastic marked this pull request as ready for review April 10, 2024 05:57

milan-elastic requested review from ishleenk17 and harnish-elastic April 10, 2024 05:57

shmsr requested changes Apr 10, 2024

View reviewed changes

milan-elastic and others added 8 commits April 10, 2024 12:15

Update readme.md

65e20e0

Co-authored-by: subham sarkar <[email protected]>

update readme.md

318d844

Co-authored-by: subham sarkar <[email protected]>

update codeowner

2221956

Merge branch 'mongodb-atlas-database-logs' of github.com:milan-elasti…

6f43d66

…c/integrations into mongodb-atlas-database-logs

update readme

43497c4

resolve review comments

cb6fc7b

update readme

fe891a7

resolve review comments

07431d8

muthu-mps reviewed Apr 15, 2024

View reviewed changes

fix ci build failure

3e248e5

add events for pipeline tests

49de0eb

milan-elastic requested review from muthu-mps and shmsr April 15, 2024 10:58

update readme.md

76ee648

resolve conflicts

297634e

milan-elastic requested a review from a team as a code owner April 16, 2024 16:50

resolve review comments

32b34ca

resolve conflicts

087e938

niraj-elastic requested changes Apr 17, 2024

View reviewed changes

shmsr requested changes Apr 17, 2024

View reviewed changes

packages/mongodb_atlas/_dev/build/docs/README.md Outdated Show resolved Hide resolved

shmsr mentioned this pull request Apr 17, 2024

[MongoDB Atlas] mongod audit datastream #9020

Merged

10 tasks

resolve review comments

e760390

milan-elastic requested review from shmsr and niraj-elastic April 18, 2024 13:09

harnish-elastic approved these changes Apr 18, 2024

View reviewed changes

niraj-elastic approved these changes Apr 18, 2024

View reviewed changes

shmsr approved these changes Apr 18, 2024

View reviewed changes

packages/mongodb_atlas/data_stream/mongod_database/fields/fields.yml Outdated Show resolved Hide resolved

packages/mongodb_atlas/data_stream/mongod_database/manifest.yml Outdated Show resolved Hide resolved

ishleenk17 approved these changes Apr 18, 2024

View reviewed changes

resolve comments

d3e36f4

milan-elastic merged commit 4558683 into elastic:main Apr 18, 2024
5 checks passed

andrewkroh added the Integration:mongodb_atlas MongoDB Atlas label Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MongoDB Atlas] Add support of mongodb database datastream #9539

[MongoDB Atlas] Add support of mongodb database datastream #9539

milan-elastic commented Apr 8, 2024 •

edited

Loading

muthu-mps left a comment

elasticmachine commented Apr 15, 2024

milan-elastic commented Apr 15, 2024

SubhrataK commented Apr 16, 2024

milan-elastic commented Apr 16, 2024

shmsr commented Apr 17, 2024 •

edited

Loading

shmsr commented Apr 17, 2024

shmsr commented Apr 18, 2024

milan-elastic commented Apr 18, 2024 •

edited

Loading

milan-elastic commented Apr 18, 2024 •

edited

Loading

shmsr commented Apr 18, 2024

shmsr commented Apr 18, 2024 •

edited

Loading

milan-elastic commented Apr 18, 2024

harnish-elastic left a comment

niraj-elastic left a comment

ishleenk17 left a comment

elastic-sonarqube bot commented Apr 18, 2024

elasticmachine commented Apr 18, 2024

elasticmachine commented Apr 18, 2024

[MongoDB Atlas] Add support of mongodb database datastream #9539

[MongoDB Atlas] Add support of mongodb database datastream #9539

Conversation

milan-elastic commented Apr 8, 2024 • edited Loading

Integration release checklist

All changes

Data dataset changes

Related Issue

Screenshots

Known Issues:

muthu-mps left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 15, 2024

🚀 Benchmarks report

milan-elastic commented Apr 15, 2024

SubhrataK commented Apr 16, 2024

milan-elastic commented Apr 16, 2024

shmsr commented Apr 17, 2024 • edited Loading

shmsr commented Apr 17, 2024

shmsr commented Apr 18, 2024

milan-elastic commented Apr 18, 2024 • edited Loading

milan-elastic commented Apr 18, 2024 • edited Loading

shmsr commented Apr 18, 2024

shmsr commented Apr 18, 2024 • edited Loading

milan-elastic commented Apr 18, 2024

harnish-elastic left a comment

Choose a reason for hiding this comment

niraj-elastic left a comment

Choose a reason for hiding this comment

ishleenk17 left a comment

Choose a reason for hiding this comment

elastic-sonarqube bot commented Apr 18, 2024

Quality Gate passed

elasticmachine commented Apr 18, 2024

💚 Build Succeeded

History

elasticmachine commented Apr 18, 2024

milan-elastic commented Apr 8, 2024 •

edited

Loading

shmsr commented Apr 17, 2024 •

edited

Loading

milan-elastic commented Apr 18, 2024 •

edited

Loading

milan-elastic commented Apr 18, 2024 •

edited

Loading

shmsr commented Apr 18, 2024 •

edited

Loading