diff --git a/packages/mongodb_atlas/_dev/build/docs/README.md b/packages/mongodb_atlas/_dev/build/docs/README.md index 1d89ec5e88c..02d14708fde 100644 --- a/packages/mongodb_atlas/_dev/build/docs/README.md +++ b/packages/mongodb_atlas/_dev/build/docs/README.md @@ -6,7 +6,7 @@ Use the MongoDB Atlas integration to: -- Collect MongoDB Audit logs and Process metrics for comprehensive monitoring and analysis. +- Collect MongoDB Atlas mongod audit logs, mongod database logs, and process metrics for comprehensive monitoring and analysis. - Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights. - Set up alerts to minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) by quickly referencing relevant logs during troubleshooting. @@ -14,17 +14,19 @@ Use the MongoDB Atlas integration to: The MongoDB Atlas integration collects logs and metrics. -Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by MongoDB Atlas integration is `mongod_audit`. +Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by MongoDB Atlas integration are `mongod_audit` and `mongod_database`. Metrics give you insight into the statistics of the MongoDB Atlas. The `Metric` data stream collected by the MongoDB Atlas integration is `process` so that the user can monitor and troubleshoot the performance of the MongoDB Atlas instance. Data streams: - `mongod_audit`: The auditing facility allows administrators and users to track system activity for deployments with multiple users and applications. Mongod Audit logs capture events related to database operations such as insertions, updates, deletions, user authentication, etc., occurring within the mongod instances. +- `mongod_database`: This datastream collects a running log of events, including entries such as incoming connections, commands run, and issues encountered. Generally, database log messages are useful for diagnosing issues, monitoring your deployment, and tuning performance. + - `process` : This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream. Note: -- Users can monitor and see the log inside the ingested documents for MongoDB Atlas in the `logs-*` index pattern from `Discover`. +- Users can monitor and see the logs and metrics inside the ingested documents for MongoDB Atlas in the `logs-*` index pattern from `Discover`. ## Prerequisites @@ -40,9 +42,11 @@ You can store and search your data using Elasticsearch and visualize and manage ### Steps to obtain Public Key, Private Key and GroupId -1. Generate programmatic API Keys with project owner permissions using the instructions in the Atlas [documentation](https://www.mongodb.com/docs/atlas/configure-api-access/#create-an-api-key-for-a-project). Then, copy the public key and private key. These serve the same function as a username and API Key respectively. -2. Enable Database Auditing for the Atlas project you want to monitor logs. You can follow the instructions provided in this Atlas [document](https://www.mongodb.com/docs/atlas/database-auditing/#procedure). -3. You can find your Project ID (Group ID) in the Atlas UI. To do this, navigate to your project, click on Settings, and copy the Project ID (Group ID). You can also programmatically find it using the Atlas Admin API or Atlas CLI as described in this Atlas [document](https://www.mongodb.com/docs/atlas/app-services/apps/metadata/#find-a-project-id). +1. Generate programmatic API keys with `project owner` permissions by following the instructions in the Atlas [documentation](https://www.mongodb.com/docs/atlas/configure-api-access/#create-an-api-key-for-a-project). Then, copy the public and private keys which function as a username and API key respectively. +2. From the Atlas UI, go to Project Settings > Access Manager > API Keys and then click on Invite To Project to add the API key created above. +3. Add specific role to API keys, under Project Settings > Access Manager > API Keys. This step is important to make sure that these API keys have the right permissions to access the data without running into any issues. The specific role for each datastream is defined under data stream reference section. +4. Enable Database Auditing for the Atlas project you want to monitor logs. You can follow the instructions provided in this Atlas [document](https://www.mongodb.com/docs/atlas/database-auditing/#procedure). +5. You can find your Project ID (Group ID) in the Atlas UI. To do this, navigate to your project, click on Settings, and copy the Project ID (Group ID). You can also programmatically find it using the Atlas Admin API or Atlas CLI as described in this Atlas [document](https://www.mongodb.com/docs/atlas/app-services/apps/metadata/#find-a-project-id). ### Important terms of MongoDB Atlas API @@ -61,7 +65,9 @@ Note: Both of above attributes can be set by using `period` in configuration par 6. Finally, save the integration. Note: -- The `mongod_audit` data stream gathers historical data spanning the previous 30 minutes. +- The `mongod_audit` and `mongod_database` data streams gather historical data spanning the previous 30 minutes. +- We recommend setting an interval of five minutes or higher for collecting mongod audit and database logs, as MongoDB Atlas refreshes logs from the cluster's backend infrastructure at five minutes intervals as described in this Atlas [document](https://www.mongodb.com/docs/atlas/reference/api-resources-spec/v2/#tag/Monitoring-and-Logs/operation/getHostLogs). +- The logs collection from MongoDB Atlas does not support M0 free clusters, M2/M5 shared clusters, or serverless instances. - Mongod: Mongod is the primary daemon method for the MongoDB system. It helps in handling the data requests, managing the data access, performing background management operations, and other core database operations. ## Troubleshooting @@ -79,16 +85,24 @@ If you encounter an error while ingesting data, it might be due to the data coll ### Mongod Audit -This is the `mongod_audit` data stream. This data stream allows administrators and users to track system activity for deployments with multiple users and applications. +This is the `mongod_audit` data stream. This data stream allows administrators and users to track system activity for deployments with multiple users and applications. To collect audit logs, the requesting API Key must have the `Project Data Access Read Only` or higher role. {{event "mongod_audit"}} {{fields "mongod_audit"}} +### Mongod Database + +This is the `mongod_database` data stream. This datastream collects a running log of events, including entries such as incoming connections, commands run, monitoring deployment, tuning performance, and issues encountered. To collect database logs, the requesting API Key must have the `Project Data Access Read Only` or higher role. + +{{event "mongod_database"}} + +{{fields "mongod_database"}} + ## Metrics reference ### Process -This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream. +This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream. To collect process metrics, the requesting API Key must have the `Project Read Only` role. {{event "process"}} diff --git a/packages/mongodb_atlas/_dev/deploy/docker/Dockerfile b/packages/mongodb_atlas/_dev/deploy/docker/Dockerfile old mode 100644 new mode 100755 diff --git a/packages/mongodb_atlas/_dev/deploy/docker/docker-compose.yml b/packages/mongodb_atlas/_dev/deploy/docker/docker-compose.yml old mode 100644 new mode 100755 diff --git a/packages/mongodb_atlas/_dev/deploy/docker/mongodb_atlas/mongod_database_data.log b/packages/mongodb_atlas/_dev/deploy/docker/mongodb_atlas/mongod_database_data.log new file mode 100755 index 00000000000..5f056906b42 --- /dev/null +++ b/packages/mongodb_atlas/_dev/deploy/docker/mongodb_atlas/mongod_database_data.log @@ -0,0 +1,5 @@ +{ "t": { "$date": "2024-03-20T19:17:06.188+00:00" }, "s": "W", "c": "CONTROL", "id": 22120, "ctx": "initandlisten", "msg": "Access control is not enabled for the database. Read and write access to data and configuration is unrestricted", "tags": [ "startupWarnings" ] } +{ "t": { "$date": "2024-02-18T14:45:23.512+00:00" }, "s": "I", "c": "NETWORK", "id": 67890, "ctx": "conn123", "msg": "Client connection accepted", "tags": [ "connection" ] } +{ "t": { "$date": "2024-02-22T10:20:05.933+00:00" }, "s": "E", "c": "STORAGE", "id": 13579, "ctx": "journal", "msg": "Journal file not found", "tags": [ "journalError" ] } +{ "t": { "$date": "2024-02-25T16:55:36.124+00:00" }, "s": "I", "c": "NETWORK", "id": 24680, "ctx": "conn456", "msg": "Client disconnected", "tags": [ "connection" ] } +{ "t": { "$date": "2024-02-28T09:12:50.007+00:00" }, "s": "E", "c": "QUERY", "id": 98765, "ctx": "queryExecutor", "msg": "Query execution failed", "tags": [ "queryError" ] } \ No newline at end of file diff --git a/packages/mongodb_atlas/changelog.yml b/packages/mongodb_atlas/changelog.yml index 10de1cb5417..14e1c43b2bb 100644 --- a/packages/mongodb_atlas/changelog.yml +++ b/packages/mongodb_atlas/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: "0.0.3" + changes: + - description: MongoDB Atlas integration package with "mongod_database" data stream. + type: enhancement + link: https://github.com/elastic/integrations/pull/9539 - version: "0.0.2" changes: - description: MongoDB Atlas integration package with "mongod_audit" data stream. diff --git a/packages/mongodb_atlas/data_stream/mongod_audit/agent/stream/input.yml.hbs b/packages/mongodb_atlas/data_stream/mongod_audit/agent/stream/input.yml.hbs index b45dff66716..ef92fe8a709 100644 --- a/packages/mongodb_atlas/data_stream/mongod_audit/agent/stream/input.yml.hbs +++ b/packages/mongodb_atlas/data_stream/mongod_audit/agent/stream/input.yml.hbs @@ -28,80 +28,72 @@ auth.digest: password: {{private_key}} resource.url: {{url}} state: - groupId: {{groupId}} + group_id: {{groupId}} want_more: false page_num: 1 redact: fields: ~ program: | - ( - ( - has(state.hostlist) && size(state.hostlist) > 0 - ? - state - : - ( - state.page_num != 1 - ? - state - : - state.with({ - "startDate": state.?cursor.last_timestamp.orValue(int(now - duration("30m"))), - "endDate": int(now) - })).as(state, state.with(request("GET", state.url + "/api/atlas/v2/groups/" + state.groupId + "/processes?pageNum=" + string(state.page_num) + "&itemsPerPage=100").with({ - "Header": { - "Accept": ["application/vnd.atlas." + string(now.getFullYear()) + "-01-01+gzip"] - } - }).do_request().as(resp, - bytes(resp.Body).decode_json().as(body, { - "hostlist": body.results.map(e, e.hostname), - "next": 0, - "page_num": body.links.exists_one(res,res.rel=="next") ? (int(state.page_num)+1) : 1 - }) - ))) - ).as(state, state.next < size(state.hostlist) - ? - ( - request("GET", state.url + "/api/atlas/v2/groups/" + state.groupId + "/clusters/" + state.hostlist[state.next] + "/logs/mongodb-audit-log.gz?startDate=" + string(int(state.startDate)) + "&endDate=" + string(int(state.endDate)) - ).with({ - "Header": { - "Accept": ["application/vnd.atlas." + string(now.getFullYear()) + "-01-01+gzip"] - } - }).do_request().as(resp, ( - resp.StatusCode == 200 && resp.ContentLength != 0 - ? - string(bytes(resp.Body).mime('application/gzip')).as(body,{ - "events": body.trim_space().split("\n").map(value,{"message":value, "host_name": state.hostlist[state.next]}), - "cursor": { - "last_timestamp": state.endDate - }, - "hostlist": (int(state.next)+1) < size(state.hostlist) ? state.hostlist : [], - "next": (int(state.next)+1) < size(state.hostlist) ? (int(state.next)+1) : 0, - "want_more": ((int(state.next)+1) < size(state.hostlist) || state.page_num != 1), - "page_num": state.page_num, - "startDate": state.startDate, - "endDate": state.endDate, - "groupId": state.groupId, - }) + ( + has(state.hostlist) && size(state.hostlist) > 0 ? + state + : + ( + state.page_num != 1 ? + state : - { - // If data is not available during this period, or if the host is unreachable, - // an event will be generated with the following message, and this event will be dropped during pipeline processing. - // keep this message in a sync with pipeline as it is being used there. - "events": [{"message":"No data for given time period or host is unreachable"}], - "cursor": { - "last_timestamp": state.endDate - }, - "hostlist": (int(state.next)+1) < size(state.hostlist) ? state.hostlist : [], - "next": (int(state.next)+1) < size(state.hostlist) ? (int(state.next)+1) : 0, - "want_more": ((int(state.next)+1) < size(state.hostlist) || state.page_num != 1), - "page_num": state.page_num, - "startDate": state.startDate, - "endDate": state.endDate, - "groupId": state.groupId, + state.with({ + "startDate": state.?cursor.last_timestamp.orValue(int(now - duration("30m"))), + "endDate": int(now) + }) + ).as(state, state.with(request("GET", state.url + "/api/atlas/v2/groups/" + state.group_id + "/processes?pageNum=" + string(state.page_num) + "&itemsPerPage=100").with({ + "Header": { + "Accept": ["application/vnd.atlas." + string(now.getFullYear()) + "-01-01+json"] + } + }).do_request().as(resp, bytes(resp.Body).decode_json().as(body, { + "hostlist": body.results.map(e, e.hostname), + "next": 0, + "page_num": body.links.exists_one(res,res.rel=="next") ? (int(state.page_num)+1) : 1 + })))) + ).as(state, state.next >= size(state.hostlist) ? {} : + ( + request("GET", state.url + "/api/atlas/v2/groups/" + state.group_id + "/clusters/" + state.hostlist[state.next] + "/logs/mongodb-audit-log.gz?startDate=" + string(int(state.startDate)) + "&endDate=" + string(int(state.endDate)) + ).with({ + "Header": { + "Accept": ["application/vnd.atlas." + string(now.getFullYear()) + "-01-01+gzip"] } - ))) - : - {} - ) - ) + }).do_request().as(resp, ( + resp.StatusCode == 200 && resp.ContentLength != 0 + ? + string(bytes(resp.Body).mime('application/gzip')).as(body,{ + "events": body.trim_space().split("\n").map(value,{"message":value, "host_name": state.hostlist[state.next]}), + "cursor": { + "last_timestamp": state.endDate + }, + "hostlist": (int(state.next)+1) < size(state.hostlist) ? state.hostlist : [], + "next": (int(state.next)+1) < size(state.hostlist) ? (int(state.next)+1) : 0, + "want_more": ((int(state.next)+1) < size(state.hostlist) || state.page_num != 1), + "page_num": state.page_num, + "startDate": state.startDate, + "endDate": state.endDate, + "group_id": state.group_id, + }) + : + { + // If data is not available during this period, or if the host is unreachable, + // an event will be generated with the following message, and this event will be dropped during pipeline processing. + // keep this message in a sync with pipeline as it is being used there. + "events": [{"message":"No data for given time period or host is unreachable"}], + "cursor": { + "last_timestamp": state.endDate + }, + "hostlist": (int(state.next)+1) < size(state.hostlist) ? state.hostlist : [], + "next": (int(state.next)+1) < size(state.hostlist) ? (int(state.next)+1) : 0, + "want_more": ((int(state.next)+1) < size(state.hostlist) || state.page_num != 1), + "page_num": state.page_num, + "startDate": state.startDate, + "endDate": state.endDate, + "group_id": state.group_id, + } + ))) + ) diff --git a/packages/mongodb_atlas/data_stream/mongod_audit/elasticsearch/ingest_pipeline/default.yml b/packages/mongodb_atlas/data_stream/mongod_audit/elasticsearch/ingest_pipeline/default.yml index 5e62d2ca2f5..57b53b0ae95 100644 --- a/packages/mongodb_atlas/data_stream/mongod_audit/elasticsearch/ingest_pipeline/default.yml +++ b/packages/mongodb_atlas/data_stream/mongod_audit/elasticsearch/ingest_pipeline/default.yml @@ -110,19 +110,31 @@ processors: field: json.uuid.$type target_field: mongodb_atlas.mongod_audit.uuid.type ignore_missing: true + - convert: + field: json.result + type: string + ignore_missing: true - script: - description: Provide information of the error code. lang: painless - ignore_failure: true + description: Maps error codes to descriptive values + tag: informative_error_code + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + params: + error_codes: + "0": "Success" + "13": "Unauthorized to perform the operation" + "18": "Authentication Failed" + "26": "Namespace Not Found" + "276": "Index build aborted" + "334": "Unauthorized to perform the operation" source: | - Map m = new HashMap(); - m.put(0, "Success"); - m.put(13, "Unauthorized to perform the operation"); - m.put(18, "Authentication Failed"); - m.put(26, "Namespace Not Found"); - m.put(276, "Index build aborted"); - m.put(334, "Unauthorized to perform the operation"); - ctx.mongodb_atlas.mongod_audit.result = m.get(ctx.json.result); + String value = ctx.json?.result; + if (value != null) { + ctx.mongodb_atlas.mongod_audit.result = params.error_codes.getOrDefault(value, null); + } - script: lang: painless source: |- @@ -139,7 +151,7 @@ processors: return false; } drop(ctx); - description: Drops null/empty values recursively. + description: Drops null and empty values recursively from the Elasticsearch document context. - remove: field: - event.original @@ -149,6 +161,7 @@ processors: field: - json ignore_missing: true + description: Removes temporary fields. - set: field: event.kind value: pipeline_error diff --git a/packages/mongodb_atlas/data_stream/mongod_audit/fields/base-fields.yml b/packages/mongodb_atlas/data_stream/mongod_audit/fields/base-fields.yml index abc42103fc8..71df896d68c 100644 --- a/packages/mongodb_atlas/data_stream/mongod_audit/fields/base-fields.yml +++ b/packages/mongodb_atlas/data_stream/mongod_audit/fields/base-fields.yml @@ -10,9 +10,6 @@ - name: input.type type: keyword description: Type of Filebeat input. -- name: tags - type: keyword - description: List of keywords used to tag each event. - name: '@timestamp' type: date description: Event timestamp. diff --git a/packages/mongodb_atlas/data_stream/mongod_audit/manifest.yml b/packages/mongodb_atlas/data_stream/mongod_audit/manifest.yml index 70b8bc2c198..1fb834b0bc7 100644 --- a/packages/mongodb_atlas/data_stream/mongod_audit/manifest.yml +++ b/packages/mongodb_atlas/data_stream/mongod_audit/manifest.yml @@ -8,15 +8,15 @@ streams: vars: - name: groupId type: text - title: GroupId - description: Identifier string that identifies your project. Groups and projects are synonymous terms. Your group id is the same as your project id. Ex. 32b6e34b3d91647abb20e7b8 + title: Group ID + description: Identifier string that identifies your project. Groups and projects are synonymous terms. Your group ID is the same as your project ID. Ex. 32b6e34b3d91647abb20e7b8 multi: false required: true show_user: true - name: interval type: text title: Interval - description: Duration between requests to the Atlas API. Supported units for this parameter are m/s. + description: Duration between requests to the Atlas API. Supported units for this parameter are m (minutes) and s (seconds). default: 5m multi: false required: true @@ -24,18 +24,18 @@ streams: - name: http_client_timeout type: text title: HTTP Client Timeout - description: Duration before declaring that the HTTP client connection has timed out. Valid time units are m/s. If the data collection period is extensive, ensure that this parameter's value is appropriately long. + description: Duration before declaring that the HTTP client connection has timed out. Valid time units are m (minutes) and s (seconds). If the data collection period is extensive, ensure that this parameter's value is appropriately long. multi: false required: true show_user: false default: 1m - name: enable_request_tracer type: bool - title: Enable request tracing + title: Enable Request Tracing multi: false required: false show_user: false - description: The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#_request_tracer_filename) for details. + description: The request tracer logs requests and responses to the agent's local file system for debugging configurations. Enabling request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_filename) for details. - name: tags type: text title: Tags @@ -47,7 +47,7 @@ streams: - name: preserve_original_event required: true show_user: true - title: Preserve original event + title: Preserve Original Event description: Preserves a raw copy of the original event, added to the field `event.original`. type: bool multi: false diff --git a/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-common-config.yml b/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-common-config.yml new file mode 100644 index 00000000000..a06dda74051 --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-common-config.yml @@ -0,0 +1,5 @@ +dynamic_fields: + "event.ingested": ".*" +fields: + tags: + - preserve_original_event diff --git a/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-mongod-database.json b/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-mongod-database.json new file mode 100644 index 00000000000..6f029249182 --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-mongod-database.json @@ -0,0 +1,20 @@ +{ + "events": [ + { + "@timestamp": "2024-02-08T06:20:49.729Z", + "message": "No data for given time period or host is unreachable" + }, + { + "@timestamp": "2024-02-08T06:20:56.621Z", + "message": "{ \"t\": { \"$date\": \"2020-05-20T19:17:06.188+00:00\" }, \"s\": \"W\", \"c\": \"CONTROL\", \"id\": 22120, \"ctx\": \"initandlisten\", \"msg\": \"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted\", \"tags\": [ \"startupWarnings\" ] }" + }, + { + "@timestamp": "2024-02-08T06:21:56.001Z", + "message": "{ \"t\": { \"$date\": \"2024-02-25T16:55:36.124+00:00\" }, \"s\": \"I\", \"c\": \"NETWORK\", \"id\": 24680, \"ctx\": \"conn456\", \"msg\": \"Client disconnected\", \"tags\": [ \"connection\" ]}" + }, + { + "@timestamp": "2024-02-08T06:22:52.121Z", + "message": "{ \"t\": { \"$date\": \"2024-02-28T09:12:50.007+00:00\" }, \"s\": \"E\", \"c\": \"QUERY\", \"id\": 98765, \"ctx\": \"queryExecutor\", \"msg\": \"Query execution failed\", \"tags\": [ \"queryError\" ]}" + } + ] +} \ No newline at end of file diff --git a/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-mongod-database.json-expected.json b/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-mongod-database.json-expected.json new file mode 100644 index 00000000000..f787dc6bf01 --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-mongod-database.json-expected.json @@ -0,0 +1,119 @@ +{ + "expected": [ + null, + { + "@timestamp": "2020-05-20T19:17:06.188Z", + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "network", + "database" + ], + "kind": "event", + "module": "mongodb_atlas", + "original": "{ \"t\": { \"$date\": \"2020-05-20T19:17:06.188+00:00\" }, \"s\": \"W\", \"c\": \"CONTROL\", \"id\": 22120, \"ctx\": \"initandlisten\", \"msg\": \"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted\", \"tags\": [ \"startupWarnings\" ] }", + "type": [ + "access", + "info" + ] + }, + "log": { + "level": "warning" + }, + "mongodb_atlas": { + "mongod_database": { + "component": "CONTROL", + "id": 22120, + "message": "Access control is not enabled for the database. Read and write access to data and configuration is unrestricted", + "tags": [ + "startupWarnings" + ], + "thread": { + "name": "initandlisten" + } + } + }, + "tags": [ + "preserve_original_event" + ] + }, + { + "@timestamp": "2024-02-25T16:55:36.124Z", + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "network", + "database" + ], + "kind": "event", + "module": "mongodb_atlas", + "original": "{ \"t\": { \"$date\": \"2024-02-25T16:55:36.124+00:00\" }, \"s\": \"I\", \"c\": \"NETWORK\", \"id\": 24680, \"ctx\": \"conn456\", \"msg\": \"Client disconnected\", \"tags\": [ \"connection\" ]}", + "type": [ + "access", + "info" + ] + }, + "log": { + "level": "informational" + }, + "mongodb_atlas": { + "mongod_database": { + "component": "NETWORK", + "id": 24680, + "message": "Client disconnected", + "tags": [ + "connection" + ], + "thread": { + "name": "conn456" + } + } + }, + "tags": [ + "preserve_original_event" + ] + }, + { + "@timestamp": "2024-02-28T09:12:50.007Z", + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "network", + "database" + ], + "kind": "event", + "module": "mongodb_atlas", + "original": "{ \"t\": { \"$date\": \"2024-02-28T09:12:50.007+00:00\" }, \"s\": \"E\", \"c\": \"QUERY\", \"id\": 98765, \"ctx\": \"queryExecutor\", \"msg\": \"Query execution failed\", \"tags\": [ \"queryError\" ]}", + "type": [ + "access", + "info" + ] + }, + "log": { + "level": "error" + }, + "mongodb_atlas": { + "mongod_database": { + "component": "QUERY", + "id": 98765, + "message": "Query execution failed", + "tags": [ + "queryError" + ], + "thread": { + "name": "queryExecutor" + } + } + }, + "tags": [ + "preserve_original_event" + ] + } + ] +} \ No newline at end of file diff --git a/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/system/test-default-config.yml b/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..19fe7ae9a87 --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/_dev/test/system/test-default-config.yml @@ -0,0 +1,13 @@ +vars: + url: + - http://{{Hostname}}:{{Port}} + public_key: + - admin + private_key: + - MongoDB@123 +data_stream: + vars: + groupId: + - mongodb-group1 +input: cel +service: mongodbatlas diff --git a/packages/mongodb_atlas/data_stream/mongod_database/agent/stream/input.yml.hbs b/packages/mongodb_atlas/data_stream/mongod_database/agent/stream/input.yml.hbs new file mode 100644 index 00000000000..8af6fd1e02f --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/agent/stream/input.yml.hbs @@ -0,0 +1,99 @@ +config_version: 2 +interval: {{interval}} +{{#if enable_request_tracer}} +resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson" +{{/if}} +{{#if ssl}} +resource.ssl: {{ssl}} +{{/if}} +{{#if http_client_timeout}} +resource.timeout: {{http_client_timeout}} +{{/if}} +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} +auth.digest: + user: {{public_key}} + password: {{private_key}} +resource.url: {{url}} +state: + group_id: {{groupId}} + want_more: false + page_num: 1 +redact: + fields: ~ +program: | + ( + has(state.hostlist) && size(state.hostlist) > 0 ? + state + : + ( + state.page_num != 1 ? + state + : + state.with({ + "startDate": state.?cursor.last_timestamp.orValue(int(now - duration("30m"))), + "endDate": int(now) + }) + ).as(state, state.with(request("GET", state.url + "/api/atlas/v2/groups/" + state.group_id + "/processes?pageNum=" + string(state.page_num) + "&itemsPerPage=100").with({ + "Header": { + "Accept": ["application/vnd.atlas." + string(now.getFullYear()) + "-01-01+json"] + } + }).do_request().as(resp, bytes(resp.Body).decode_json().as(body, { + "hostlist": body.results.map(e, e.hostname), + "next": 0, + "page_num": body.links.exists_one(res,res.rel=="next") ? (int(state.page_num)+1) : 1 + })))) + ).as(state, state.next >= size(state.hostlist) ? {} : + ( + request("GET", state.url + "/api/atlas/v2/groups/" + state.group_id + "/clusters/" + state.hostlist[state.next] + "/logs/mongodb.gz?startDate=" + string(int(state.startDate)) + "&endDate=" + string(int(state.endDate)) + ).with({ + "Header": { + "Accept": ["application/vnd.atlas." + string(now.getFullYear()) + "-01-01+gzip"] + } + }).do_request().as(resp, ( + resp.StatusCode == 200 && resp.ContentLength != 0 + ? + string(bytes(resp.Body).mime('application/gzip')).as(body,{ + "events": body.trim_space().split("\n").map(value,{"message":value, "host_name": state.hostlist[state.next]}), + "cursor": { + "last_timestamp": state.endDate + }, + "hostlist": (int(state.next)+1) < size(state.hostlist) ? state.hostlist : [], + "next": (int(state.next)+1) < size(state.hostlist) ? (int(state.next)+1) : 0, + "want_more": ((int(state.next)+1) < size(state.hostlist) || state.page_num != 1), + "page_num": state.page_num, + "startDate": state.startDate, + "endDate": state.endDate, + "group_id": state.group_id, + }) + : + { + // If data is not available during this period, or if the host is unreachable, + // an event will be generated with the following message, and this event will be dropped during pipeline processing. + // keep this message in a sync with pipeline as it is being used there. + "events": [{"message":"No data for given time period or host is unreachable"}], + "cursor": { + "last_timestamp": state.endDate + }, + "hostlist": (int(state.next)+1) < size(state.hostlist) ? state.hostlist : [], + "next": (int(state.next)+1) < size(state.hostlist) ? (int(state.next)+1) : 0, + "want_more": ((int(state.next)+1) < size(state.hostlist) || state.page_num != 1), + "page_num": state.page_num, + "startDate": state.startDate, + "endDate": state.endDate, + "group_id": state.group_id, + } + ))) + ) diff --git a/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml b/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..b8b9cfb563a --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,142 @@ +--- +description: Pipeline for processing MongoDB Atlas Mongod Database logs. +processors: + - drop: + if: ctx.message == "No data for given time period or host is unreachable" + tag: drop_empty_events + - set: + field: ecs.version + value: 8.11.0 + - set: + field: event.kind + value: event + - set: + field: event.module + value: mongodb_atlas + - set: + field: event.category + value: ["network", "database"] + - set: + field: event.type + value: ["access", "info"] + - rename: + field: message + target_field: event.original + ignore_missing: true + if: 'ctx.event?.original == null' + - rename: + field: host_name + target_field: mongodb_atlas.mongod_database.hostname + ignore_missing: true + - json: + field: event.original + target_field: json + tag: json_decoding + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - date: + field: json.t.$date + target_field: '@timestamp' + formats: [ISO8601] + on_failure: + - append: + field: error.message + value: '{{ _ingest.on_failure_message }}' + - script: + lang: painless + description: Maps log severity levels to descriptive values + tag: informative_log_level + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + params: + severity_levels: + F: fatal + E: error + W: warning + I: informational + D1: debug 1 + D2: debug 2 + D3: debug 3 + D4: debug 4 + D5: debug 5 + source: | + String value = ctx.json?.s; + if (value != null) { + ctx.severity = params.severity_levels.getOrDefault(value, null); + } + - rename: + field: severity + target_field: log.level + ignore_missing: true + - rename: + field: json.msg + target_field: mongodb_atlas.mongod_database.message + ignore_missing: true + - rename: + field: json.c + target_field: mongodb_atlas.mongod_database.component + ignore_missing: true + - rename: + field: json.id + target_field: mongodb_atlas.mongod_database.id + ignore_missing: true + - rename: + field: json.ctx + target_field: mongodb_atlas.mongod_database.thread.name + ignore_missing: true + - rename: + field: json.attr + target_field: mongodb_atlas.mongod_database.attributes + ignore_missing: true + - rename: + field: json.tags + target_field: mongodb_atlas.mongod_database.tags + ignore_missing: true + - rename: + field: json.size + target_field: mongodb_atlas.mongod_database.size + ignore_missing: true + - script: + lang: painless + source: |- + boolean drop(Object o) { + if (o == null || o == '') { + return true; + } else if (o instanceof Map) { + ((Map) o).values().removeIf(v -> drop(v)); + return (((Map) o).size() == 0); + } else if (o instanceof List) { + ((List) o).removeIf(v -> drop(v)); + return (((List) o).length == 0); + } + return false; + } + drop(ctx); + description: Drops null and empty values recursively from the Elasticsearch document context. + - remove: + field: + - event.original + tag: remove_original_event + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event')) + - remove: + field: + - json + ignore_missing: true + description: Removes temporary fields. + - set: + field: event.kind + value: pipeline_error + if: ctx.error?.message != null +on_failure: + - append: + field: error.message + value: '{{{ _ingest.on_failure_message }}}' + - append: + field: event.kind + value: pipeline_error + allow_duplicates: false diff --git a/packages/mongodb_atlas/data_stream/mongod_database/fields/base-fields.yml b/packages/mongodb_atlas/data_stream/mongod_database/fields/base-fields.yml new file mode 100644 index 00000000000..71df896d68c --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/fields/base-fields.yml @@ -0,0 +1,15 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: input.type + type: keyword + description: Type of Filebeat input. +- name: '@timestamp' + type: date + description: Event timestamp. diff --git a/packages/mongodb_atlas/data_stream/mongod_database/fields/fields.yml b/packages/mongodb_atlas/data_stream/mongod_database/fields/fields.yml new file mode 100644 index 00000000000..06bd20b95f9 --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/fields/fields.yml @@ -0,0 +1,33 @@ +- name: mongodb_atlas.mongod_database + type: group + fields: + - name: attributes + type: object + object_type: keyword + description: One or more key-value pairs for additional log attributes. If a log message does not include any additional attributes, the attr object is omitted. + - name: component + type: keyword + description: The component field indicates the category to which a logged event belongs, such as NETWORK or COMMAND. + - name: hostname + type: keyword + description: A human-readable label that identifies the host that stores the log files you want to download. + - name: id + type: long + description: The unique identifier for the log statement. + - name: message + type: match_only_text + description: The log output message passed from the server or driver. If necessary, the message is escaped according to the JSON specification. + - name: size + type: object + object_type: keyword + description: The original size of a log entry if it has been truncated. Only included if the log entry contains at least one truncated attr attribute. + - name: tags + type: keyword + description: Strings representing any tags applicable to the log statement, for example, ["startupWarnings"]. + - name: thread.name + type: keyword + description: The name of the thread that caused the log statement. + - name: truncated + type: object + object_type: keyword + description: Information about log message truncation, if applicable. Only included if the log entry contains at least one truncated attr attribute. diff --git a/packages/mongodb_atlas/data_stream/mongod_database/manifest.yml b/packages/mongodb_atlas/data_stream/mongod_database/manifest.yml new file mode 100644 index 00000000000..56a35cf9b74 --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/manifest.yml @@ -0,0 +1,62 @@ +title: Collect Mongod Database logs from MongoDB Atlas +type: logs +streams: + - input: cel + title: MongoDB Atlas Mongod Database logs + description: Collect Mongod Database logs from MongoDB Atlas. + template_path: input.yml.hbs + vars: + - name: groupId + type: text + title: Group ID + description: Identifier string that identifies your project. Groups and projects are synonymous terms. Your group ID is the same as your project ID. Ex. 32b6e34b3d91647abb20e7b8 + multi: false + required: true + show_user: true + - name: interval + type: text + title: Interval + description: Duration between requests to the Atlas API. Supported units for this parameter are m (minutes) and s (seconds). + default: 5m + multi: false + required: true + show_user: false + - name: http_client_timeout + type: text + title: HTTP Client Timeout + description: Duration before declaring that the HTTP client connection has timed out. Valid time units are m (minutes) and s (seconds). If the data collection period is extensive, ensure that this parameter's value is appropriately long. + multi: false + required: true + show_user: false + default: 1m + - name: enable_request_tracer + type: bool + title: Enable Request Tracing + multi: false + required: false + show_user: false + description: The request tracer logs requests and responses to the agent's local file system for debugging configurations. Enabling request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_filename) for details. + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - mongodb_atlas-mongod_database + - name: preserve_original_event + required: true + show_user: true + title: Preserve Original Event + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html) for details. diff --git a/packages/mongodb_atlas/data_stream/mongod_database/sample_event.json b/packages/mongodb_atlas/data_stream/mongod_database/sample_event.json new file mode 100644 index 00000000000..518d71baccd --- /dev/null +++ b/packages/mongodb_atlas/data_stream/mongod_database/sample_event.json @@ -0,0 +1,83 @@ +{ + "@timestamp": "2024-02-18T14:45:23.512Z", + "agent": { + "ephemeral_id": "dbad1b64-5ae2-467e-a76a-7d31d2bbc35a", + "id": "35b61223-ca83-481f-a4aa-ab5983a75ba8", + "name": "docker-fleet-agent", + "type": "filebeat", + "version": "8.13.0" + }, + "data_stream": { + "dataset": "mongodb_atlas.mongod_database", + "namespace": "ep", + "type": "logs" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "35b61223-ca83-481f-a4aa-ab5983a75ba8", + "snapshot": false, + "version": "8.13.0" + }, + "event": { + "agent_id_status": "verified", + "category": [ + "network", + "database" + ], + "dataset": "mongodb_atlas.mongod_database", + "ingested": "2024-04-05T10:24:59Z", + "kind": "event", + "module": "mongodb_atlas", + "type": [ + "access", + "info" + ] + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "docker-fleet-agent", + "id": "8259e024976a406e8a54cdbffeb84fec", + "ip": [ + "192.168.255.7" + ], + "mac": [ + "02-42-C0-A8-FF-07" + ], + "name": "docker-fleet-agent", + "os": { + "codename": "focal", + "family": "debian", + "kernel": "3.10.0-1160.92.1.el7.x86_64", + "name": "Ubuntu", + "platform": "ubuntu", + "type": "linux", + "version": "20.04.6 LTS (Focal Fossa)" + } + }, + "input": { + "type": "cel" + }, + "log": { + "level": "informational" + }, + "mongodb_atlas": { + "mongod_database": { + "component": "NETWORK", + "hostname": "hostname1", + "id": 67890, + "message": "Client connection accepted", + "tags": [ + "connection" + ], + "thread": { + "name": "conn123" + } + } + }, + "tags": [ + "mongodb_atlas-mongod_database" + ] +} \ No newline at end of file diff --git a/packages/mongodb_atlas/docs/README.md b/packages/mongodb_atlas/docs/README.md index cc17df46cd8..306cb032603 100644 --- a/packages/mongodb_atlas/docs/README.md +++ b/packages/mongodb_atlas/docs/README.md @@ -6,7 +6,7 @@ Use the MongoDB Atlas integration to: -- Collect MongoDB Audit logs and Process metrics for comprehensive monitoring and analysis. +- Collect MongoDB Atlas mongod audit logs, mongod database logs, and process metrics for comprehensive monitoring and analysis. - Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights. - Set up alerts to minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) by quickly referencing relevant logs during troubleshooting. @@ -14,17 +14,19 @@ Use the MongoDB Atlas integration to: The MongoDB Atlas integration collects logs and metrics. -Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by MongoDB Atlas integration is `mongod_audit`. +Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by MongoDB Atlas integration are `mongod_audit` and `mongod_database`. Metrics give you insight into the statistics of the MongoDB Atlas. The `Metric` data stream collected by the MongoDB Atlas integration is `process` so that the user can monitor and troubleshoot the performance of the MongoDB Atlas instance. Data streams: - `mongod_audit`: The auditing facility allows administrators and users to track system activity for deployments with multiple users and applications. Mongod Audit logs capture events related to database operations such as insertions, updates, deletions, user authentication, etc., occurring within the mongod instances. +- `mongod_database`: This datastream collects a running log of events, including entries such as incoming connections, commands run, and issues encountered. Generally, database log messages are useful for diagnosing issues, monitoring your deployment, and tuning performance. + - `process` : This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream. Note: -- Users can monitor and see the log inside the ingested documents for MongoDB Atlas in the `logs-*` index pattern from `Discover`. +- Users can monitor and see the logs and metrics inside the ingested documents for MongoDB Atlas in the `logs-*` index pattern from `Discover`. ## Prerequisites @@ -40,9 +42,11 @@ You can store and search your data using Elasticsearch and visualize and manage ### Steps to obtain Public Key, Private Key and GroupId -1. Generate programmatic API Keys with project owner permissions using the instructions in the Atlas [documentation](https://www.mongodb.com/docs/atlas/configure-api-access/#create-an-api-key-for-a-project). Then, copy the public key and private key. These serve the same function as a username and API Key respectively. -2. Enable Database Auditing for the Atlas project you want to monitor logs. You can follow the instructions provided in this Atlas [document](https://www.mongodb.com/docs/atlas/database-auditing/#procedure). -3. You can find your Project ID (Group ID) in the Atlas UI. To do this, navigate to your project, click on Settings, and copy the Project ID (Group ID). You can also programmatically find it using the Atlas Admin API or Atlas CLI as described in this Atlas [document](https://www.mongodb.com/docs/atlas/app-services/apps/metadata/#find-a-project-id). +1. Generate programmatic API keys with `project owner` permissions by following the instructions in the Atlas [documentation](https://www.mongodb.com/docs/atlas/configure-api-access/#create-an-api-key-for-a-project). Then, copy the public and private keys which function as a username and API key respectively. +2. From the Atlas UI, go to Project Settings > Access Manager > API Keys and then click on Invite To Project to add the API key created above. +3. Add specific role to API keys, under Project Settings > Access Manager > API Keys. This step is important to make sure that these API keys have the right permissions to access the data without running into any issues. The specific role for each datastream is defined under data stream reference section. +4. Enable Database Auditing for the Atlas project you want to monitor logs. You can follow the instructions provided in this Atlas [document](https://www.mongodb.com/docs/atlas/database-auditing/#procedure). +5. You can find your Project ID (Group ID) in the Atlas UI. To do this, navigate to your project, click on Settings, and copy the Project ID (Group ID). You can also programmatically find it using the Atlas Admin API or Atlas CLI as described in this Atlas [document](https://www.mongodb.com/docs/atlas/app-services/apps/metadata/#find-a-project-id). ### Important terms of MongoDB Atlas API @@ -61,7 +65,9 @@ Note: Both of above attributes can be set by using `period` in configuration par 6. Finally, save the integration. Note: -- The `mongod_audit` data stream gathers historical data spanning the previous 30 minutes. +- The `mongod_audit` and `mongod_database` data streams gather historical data spanning the previous 30 minutes. +- We recommend setting an interval of five minutes or higher for collecting mongod audit and database logs, as MongoDB Atlas refreshes logs from the cluster's backend infrastructure at five minutes intervals as described in this Atlas [document](https://www.mongodb.com/docs/atlas/reference/api-resources-spec/v2/#tag/Monitoring-and-Logs/operation/getHostLogs). +- The logs collection from MongoDB Atlas does not support M0 free clusters, M2/M5 shared clusters, or serverless instances. - Mongod: Mongod is the primary daemon method for the MongoDB system. It helps in handling the data requests, managing the data access, performing background management operations, and other core database operations. ## Troubleshooting @@ -79,7 +85,7 @@ If you encounter an error while ingesting data, it might be due to the data coll ### Mongod Audit -This is the `mongod_audit` data stream. This data stream allows administrators and users to track system activity for deployments with multiple users and applications. +This is the `mongod_audit` data stream. This data stream allows administrators and users to track system activity for deployments with multiple users and applications. To collect audit logs, the requesting API Key must have the `Project Data Access Read Only` or higher role. An example event for `mongod_audit` looks as following: @@ -215,13 +221,124 @@ An example event for `mongod_audit` looks as following: | mongodb_atlas.mongod_audit.user.roles | Array of documents that specify the roles granted to the user. | object | | mongodb_atlas.mongod_audit.uuid.binary | Document that contains a universally unique identifier (UUID) for the audit message. | keyword | | mongodb_atlas.mongod_audit.uuid.type | The $type field specifies the BSON subtype for the $binary field. | keyword | -| tags | List of keywords used to tag each event. | keyword | + + +### Mongod Database + +This is the `mongod_database` data stream. This datastream collects a running log of events, including entries such as incoming connections, commands run, monitoring deployment, tuning performance, and issues encountered. To collect database logs, the requesting API Key must have the `Project Data Access Read Only` or higher role. + +An example event for `mongod_database` looks as following: + +```json +{ + "@timestamp": "2024-02-18T14:45:23.512Z", + "agent": { + "ephemeral_id": "dbad1b64-5ae2-467e-a76a-7d31d2bbc35a", + "id": "35b61223-ca83-481f-a4aa-ab5983a75ba8", + "name": "docker-fleet-agent", + "type": "filebeat", + "version": "8.13.0" + }, + "data_stream": { + "dataset": "mongodb_atlas.mongod_database", + "namespace": "ep", + "type": "logs" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "35b61223-ca83-481f-a4aa-ab5983a75ba8", + "snapshot": false, + "version": "8.13.0" + }, + "event": { + "agent_id_status": "verified", + "category": [ + "network", + "database" + ], + "dataset": "mongodb_atlas.mongod_database", + "ingested": "2024-04-05T10:24:59Z", + "kind": "event", + "module": "mongodb_atlas", + "type": [ + "access", + "info" + ] + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "docker-fleet-agent", + "id": "8259e024976a406e8a54cdbffeb84fec", + "ip": [ + "192.168.255.7" + ], + "mac": [ + "02-42-C0-A8-FF-07" + ], + "name": "docker-fleet-agent", + "os": { + "codename": "focal", + "family": "debian", + "kernel": "3.10.0-1160.92.1.el7.x86_64", + "name": "Ubuntu", + "platform": "ubuntu", + "type": "linux", + "version": "20.04.6 LTS (Focal Fossa)" + } + }, + "input": { + "type": "cel" + }, + "log": { + "level": "informational" + }, + "mongodb_atlas": { + "mongod_database": { + "component": "NETWORK", + "hostname": "hostname1", + "id": 67890, + "message": "Client connection accepted", + "tags": [ + "connection" + ], + "thread": { + "name": "conn123" + } + } + }, + "tags": [ + "mongodb_atlas-mongod_database" + ] +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| input.type | Type of Filebeat input. | keyword | +| mongodb_atlas.mongod_database.attributes | One or more key-value pairs for additional log attributes. If a log message does not include any additional attributes, the attr object is omitted. | object | +| mongodb_atlas.mongod_database.component | The component field indicates the category to which a logged event belongs, such as NETWORK or COMMAND. | keyword | +| mongodb_atlas.mongod_database.hostname | A human-readable label that identifies the host that stores the log files you want to download. | keyword | +| mongodb_atlas.mongod_database.id | The unique identifier for the log statement. | long | +| mongodb_atlas.mongod_database.message | The log output message passed from the server or driver. If necessary, the message is escaped according to the JSON specification. | match_only_text | +| mongodb_atlas.mongod_database.size | The original size of a log entry if it has been truncated. Only included if the log entry contains at least one truncated attr attribute. | object | +| mongodb_atlas.mongod_database.tags | Strings representing any tags applicable to the log statement, for example, ["startupWarnings"]. | keyword | +| mongodb_atlas.mongod_database.thread.name | The name of the thread that caused the log statement. | keyword | +| mongodb_atlas.mongod_database.truncated | Information about log message truncation, if applicable. Only included if the log entry contains at least one truncated attr attribute. | object | ## Metrics reference ### Process -This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream. +This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream. To collect process metrics, the requesting API Key must have the `Project Read Only` role. An example event for `process` looks as following: diff --git a/packages/mongodb_atlas/img/mongodb_atlas-mongod-database-dashboard.png b/packages/mongodb_atlas/img/mongodb_atlas-mongod-database-dashboard.png new file mode 100644 index 00000000000..d82242b7a15 Binary files /dev/null and b/packages/mongodb_atlas/img/mongodb_atlas-mongod-database-dashboard.png differ diff --git a/packages/mongodb_atlas/kibana/dashboard/mongodb_atlas-b6ceb5eb-c380-42c1-a3ca-8fcd0bc3dc50.json b/packages/mongodb_atlas/kibana/dashboard/mongodb_atlas-b6ceb5eb-c380-42c1-a3ca-8fcd0bc3dc50.json new file mode 100644 index 00000000000..55568a8e0b3 --- /dev/null +++ b/packages/mongodb_atlas/kibana/dashboard/mongodb_atlas-b6ceb5eb-c380-42c1-a3ca-8fcd0bc3dc50.json @@ -0,0 +1,951 @@ +{ + "attributes": { + "controlGroupInput": { + "chainingSystem": "HIERARCHICAL", + "controlStyle": "oneLine", + "ignoreParentSettingsJSON": "{\"ignoreFilters\":false,\"ignoreQuery\":false,\"ignoreTimerange\":false,\"ignoreValidations\":false}", + "panelsJSON": "{\"b1c83e14-fc43-4648-8f9c-1a0e87595360\":{\"type\":\"optionsListControl\",\"order\":0,\"grow\":false,\"width\":\"medium\",\"explicitInput\":{\"id\":\"b1c83e14-fc43-4648-8f9c-1a0e87595360\",\"fieldName\":\"mongodb_atlas.mongod_database.hostname\",\"title\":\"Hostname\",\"grow\":true,\"width\":\"medium\",\"enhancements\":{}}}}" + }, + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "mongodb_atlas.mongod_database" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "mongodb_atlas.mongod_database" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-25f87099-4fb8-427d-b16b-ebd64f4a85f9", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "eb47556c-d770-4cc4-adba-5bc26a9c99e9", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "fe22ce24-cb25-4bfc-b216-f24615c19601", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "25f87099-4fb8-427d-b16b-ebd64f4a85f9": { + "columnOrder": [ + "30fa29b6-0489-4439-9abe-f81510cec2d8", + "2ea13e2d-1331-4764-a604-70ff1d43bbe6", + "87c7bf6b-2c71-4494-b667-a8e09ecba76a" + ], + "columns": { + "2ea13e2d-1331-4764-a604-70ff1d43bbe6": { + "dataType": "string", + "isBucketed": true, + "label": "Top 5 values of log.level", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "87c7bf6b-2c71-4494-b667-a8e09ecba76a", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "log.level" + }, + "30fa29b6-0489-4439-9abe-f81510cec2d8": { + "dataType": "date", + "isBucketed": true, + "label": "@timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": false, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "@timestamp" + }, + "87c7bf6b-2c71-4494-b667-a8e09ecba76a": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "eb47556c-d770-4cc4-adba-5bc26a9c99e9", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "mongodb_atlas.mongod_database" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "mongodb_atlas.mongod_database" + } + } + }, + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "log.level", + "index": "fe22ce24-cb25-4bfc-b216-f24615c19601", + "key": "log.level", + "negate": true, + "params": { + "query": "informational" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "log.level": "informational" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "curveType": "LINEAR", + "emphasizeFitting": true, + "endValue": "None", + "fittingFunction": "Linear", + "layers": [ + { + "accessors": [ + "87c7bf6b-2c71-4494-b667-a8e09ecba76a" + ], + "layerId": "25f87099-4fb8-427d-b16b-ebd64f4a85f9", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "splitAccessor": "2ea13e2d-1331-4764-a604-70ff1d43bbe6", + "xAccessor": "30fa29b6-0489-4439-9abe-f81510cec2d8" + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "line", + "title": "Empty XY chart", + "valueLabels": "hide", + "valuesInLegend": true + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {}, + "hidePanelTitles": false + }, + "gridData": { + "h": 13, + "i": "556840e3-fd50-40ee-8dac-81fb08a9e2f4", + "w": 48, + "x": 0, + "y": 0 + }, + "panelIndex": "556840e3-fd50-40ee-8dac-81fb08a9e2f4", + "title": "Severity over time [Logs MongoDB Atlas]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-de3eac83-7fe6-437f-a2a6-e68ac19cb300", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "de3eac83-7fe6-437f-a2a6-e68ac19cb300": { + "columnOrder": [ + "32af9b6f-edd4-4102-9e28-3a1bb7e778dc", + "0a509068-0967-410b-b1df-b3f5e0847d40" + ], + "columns": { + "0a509068-0967-410b-b1df-b3f5e0847d40": { + "dataType": "number", + "isBucketed": false, + "label": "Count of log.level", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "log.level" + }, + "32af9b6f-edd4-4102-9e28-3a1bb7e778dc": { + "dataType": "string", + "isBucketed": true, + "label": "Top 5 values of mongodb_atlas.mongod_database.component", + "operationType": "terms", + "params": { + "accuracyMode": true, + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "0a509068-0967-410b-b1df-b3f5e0847d40", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "mongodb_atlas.mongod_database.component" + } + }, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "dbcbe9e4-d0eb-400e-a756-1e331bc5fe82", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "mongodb_atlas.mongod_database" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "mongodb_atlas.mongod_database" + } + } + }, + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "log.level", + "index": "1a12e5ae-d0db-4c64-92bf-6ef751aac1e4", + "key": "log.level", + "negate": false, + "params": { + "query": "error" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "log.level": "error" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "emptySizeRatio": 0.3, + "layerId": "de3eac83-7fe6-437f-a2a6-e68ac19cb300", + "layerType": "data", + "legendDisplay": "default", + "legendPosition": "right", + "metrics": [ + "0a509068-0967-410b-b1df-b3f5e0847d40" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "32af9b6f-edd4-4102-9e28-3a1bb7e778dc" + ] + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": {}, + "hidePanelTitles": false + }, + "gridData": { + "h": 18, + "i": "a6febfb1-be15-4e8d-97df-d7c898cc0aff", + "w": 16, + "x": 0, + "y": 26 + }, + "panelIndex": "a6febfb1-be15-4e8d-97df-d7c898cc0aff", + "title": "Breakdown of error by components [Logs MongoDB Atlas]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-27ceb5f7-3fd0-4bb5-bcb8-02c056e4e0a5", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "6ec257bf-90cf-4ab5-a72c-c26208d0ebcd", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "27ceb5f7-3fd0-4bb5-bcb8-02c056e4e0a5": { + "columnOrder": [ + "183e55cf-a8d9-42f7-b743-0e1043f0632e", + "3bb6761c-6cdb-462e-bad1-0da722838a70" + ], + "columns": { + "183e55cf-a8d9-42f7-b743-0e1043f0632e": { + "dataType": "string", + "isBucketed": true, + "label": "Top 5 values of log.level", + "operationType": "terms", + "params": { + "accuracyMode": true, + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "3bb6761c-6cdb-462e-bad1-0da722838a70", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "log.level" + }, + "3bb6761c-6cdb-462e-bad1-0da722838a70": { + "dataType": "number", + "isBucketed": false, + "label": "Count of records", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "6ec257bf-90cf-4ab5-a72c-c26208d0ebcd", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "mongodb_atlas.mongod_database" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "mongodb_atlas.mongod_database" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "layerId": "27ceb5f7-3fd0-4bb5-bcb8-02c056e4e0a5", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "3bb6761c-6cdb-462e-bad1-0da722838a70" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "183e55cf-a8d9-42f7-b743-0e1043f0632e" + ] + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": {}, + "hidePanelTitles": false + }, + "gridData": { + "h": 18, + "i": "fdb9a1a3-a892-49cf-87ef-ed2311b88040", + "w": 16, + "x": 16, + "y": 26 + }, + "panelIndex": "fdb9a1a3-a892-49cf-87ef-ed2311b88040", + "title": "Breakdown of severity [Logs MongoDB Atlas]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-96d79b7e-3e2c-4d1f-9068-e0343ed98ec8", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "96d79b7e-3e2c-4d1f-9068-e0343ed98ec8": { + "columnOrder": [ + "777322ed-4baa-48db-b611-4127a3c7863e", + "c5277f20-8ba5-45ad-9077-50c4c4b3286d" + ], + "columns": { + "777322ed-4baa-48db-b611-4127a3c7863e": { + "dataType": "string", + "isBucketed": true, + "label": "Top 5 values of mongodb_atlas.mongod_database.component", + "operationType": "terms", + "params": { + "accuracyMode": true, + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "c5277f20-8ba5-45ad-9077-50c4c4b3286d", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "secondaryFields": [], + "size": 5 + }, + "scale": "ordinal", + "sourceField": "mongodb_atlas.mongod_database.component" + }, + "c5277f20-8ba5-45ad-9077-50c4c4b3286d": { + "dataType": "number", + "isBucketed": false, + "label": "Count of log.level", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "log.level" + } + }, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "7f33123a-e17f-4b57-9f57-72b0687a47b5", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "mongodb_atlas.mongod_database" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "mongodb_atlas.mongod_database" + } + } + }, + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "log.level", + "index": "cbf8633c-f667-49c5-9e87-9bc929392f10", + "key": "log.level", + "negate": false, + "params": { + "query": "warning" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "log.level": "warning" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "layerId": "96d79b7e-3e2c-4d1f-9068-e0343ed98ec8", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "c5277f20-8ba5-45ad-9077-50c4c4b3286d" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "777322ed-4baa-48db-b611-4127a3c7863e" + ] + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": {}, + "hidePanelTitles": false + }, + "gridData": { + "h": 18, + "i": "3dc2c4c6-e34a-41ce-860d-7e0700e6deec", + "w": 16, + "x": 32, + "y": 26 + }, + "panelIndex": "3dc2c4c6-e34a-41ce-860d-7e0700e6deec", + "title": "Breakdown of warning by components [Logs MongoDB Atlas]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-5d61c1b6-6caa-44f6-bdfe-56af89106f60", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "ee5bc37c-3b8f-446f-84d7-83f75074598c", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "5d61c1b6-6caa-44f6-bdfe-56af89106f60": { + "columnOrder": [ + "04b5cda6-8620-4be0-9544-fab71ceff0e0", + "1203f700-d9eb-416b-8a4d-ae30107aa93e", + "11295859-7d82-4726-bd06-0c34ddc4d91b" + ], + "columns": { + "04b5cda6-8620-4be0-9544-fab71ceff0e0": { + "dataType": "string", + "isBucketed": true, + "label": "Top 5 values of mongodb_atlas.mongod_database.hostname", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "11295859-7d82-4726-bd06-0c34ddc4d91b", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "mongodb_atlas.mongod_database.hostname" + }, + "11295859-7d82-4726-bd06-0c34ddc4d91b": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": true + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "1203f700-d9eb-416b-8a4d-ae30107aa93e": { + "dataType": "date", + "isBucketed": true, + "label": "@timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": false, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "@timestamp" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "ee5bc37c-3b8f-446f-84d7-83f75074598c", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "mongodb_atlas.mongod_database" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "mongodb_atlas.mongod_database" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "curveType": "LINEAR", + "emphasizeFitting": true, + "endValue": "None", + "fittingFunction": "Linear", + "layers": [ + { + "accessors": [ + "11295859-7d82-4726-bd06-0c34ddc4d91b" + ], + "layerId": "5d61c1b6-6caa-44f6-bdfe-56af89106f60", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "splitAccessor": "04b5cda6-8620-4be0-9544-fab71ceff0e0", + "xAccessor": "1203f700-d9eb-416b-8a4d-ae30107aa93e" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "line", + "title": "Empty XY chart", + "valueLabels": "hide", + "valuesInLegend": true + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {}, + "hidePanelTitles": false + }, + "gridData": { + "h": 13, + "i": "6d436f31-838a-497b-9b83-bfd3cf66f23c", + "w": 48, + "x": 0, + "y": 13 + }, + "panelIndex": "6d436f31-838a-497b-9b83-bfd3cf66f23c", + "title": "Log messages by host over time [Logs MongoDB Atlas]", + "type": "lens" + } + ], + "timeRestore": false, + "title": "[Logs MongoDB Atlas] Mongod Database", + "version": 1 + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2024-04-16T13:16:31.306Z", + "id": "mongodb_atlas-b6ceb5eb-c380-42c1-a3ca-8fcd0bc3dc50", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "556840e3-fd50-40ee-8dac-81fb08a9e2f4:indexpattern-datasource-layer-25f87099-4fb8-427d-b16b-ebd64f4a85f9", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "556840e3-fd50-40ee-8dac-81fb08a9e2f4:eb47556c-d770-4cc4-adba-5bc26a9c99e9", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "556840e3-fd50-40ee-8dac-81fb08a9e2f4:fe22ce24-cb25-4bfc-b216-f24615c19601", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "a6febfb1-be15-4e8d-97df-d7c898cc0aff:indexpattern-datasource-layer-de3eac83-7fe6-437f-a2a6-e68ac19cb300", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "fdb9a1a3-a892-49cf-87ef-ed2311b88040:indexpattern-datasource-layer-27ceb5f7-3fd0-4bb5-bcb8-02c056e4e0a5", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "fdb9a1a3-a892-49cf-87ef-ed2311b88040:6ec257bf-90cf-4ab5-a72c-c26208d0ebcd", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "3dc2c4c6-e34a-41ce-860d-7e0700e6deec:indexpattern-datasource-layer-96d79b7e-3e2c-4d1f-9068-e0343ed98ec8", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "6d436f31-838a-497b-9b83-bfd3cf66f23c:indexpattern-datasource-layer-5d61c1b6-6caa-44f6-bdfe-56af89106f60", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "6d436f31-838a-497b-9b83-bfd3cf66f23c:ee5bc37c-3b8f-446f-84d7-83f75074598c", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_b1c83e14-fc43-4648-8f9c-1a0e87595360:optionsListDataView", + "type": "index-pattern" + } + ], + "type": "dashboard", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/mongodb_atlas/manifest.yml b/packages/mongodb_atlas/manifest.yml index 0d4cdce6d3e..5e0f723e8a4 100644 --- a/packages/mongodb_atlas/manifest.yml +++ b/packages/mongodb_atlas/manifest.yml @@ -1,7 +1,7 @@ format_version: 3.0.3 name: mongodb_atlas title: "MongoDB Atlas" -version: 0.0.2 +version: 0.0.3 source: license: "Elastic-2.0" description: This Elastic integration collects logs and metrics from MongoDB Atlas instance. @@ -15,6 +15,10 @@ conditions: elastic: subscription: "basic" screenshots: + - src: /img/mongodb_atlas-mongod-database-dashboard.png + title: MongoDB Atlas Mongod Database Dashboard + size: 600x600 + type: image/png - src: /img/mongodb_atlas-mongod-audit-dashboard.png title: MongoDB Atlas Mongod Audit Dashboard size: 600x600