Mongodb Atlas receiver not able to pull logs issue #28851

ammarasheikh · 2023-11-01T18:15:56Z

Component(s)

receiver/mongodbatlas

Describe the issue you're reporting

Hi,

Sorry for my lack of knowledge as I am using otelemetry mongodb atlas reciever for the first time. I am using the example yaml configuration as below: but I am keep receiving this warning and not able to retrieve any logs. I have the API keys having or owner permission. Not quite sure why I am not seeing any logs. Please help. Thanks

2023-11-01T18:11:20.383Z warn [email protected]/logs.go:229 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos.gz"} ││ 2023-11-01T18:11:20.687Z warn [email protected]/logs.go:259 Attempted to retrieve audit logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos-aud │

Here's my config:
receivers:
mongodbatlas:
# existing metrics settings
public_key: ${MONGODB_ATLAS_PUBLIC_KEY}
private_key: ${MONGODB_ATLAS_PRIVATE_KEY}
granularity: PT1M
retry_on_failure:
enabled: true
# new logging settings
logs:
enabled: true #default true
projects: # List of projects to monitor logs from
- name: project_1 # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true# The user needs to enable this in Atlas so it needs to be enabled here

github-actions · 2023-11-01T18:16:18Z

Pinging code owners:

receiver/mongodbatlas: @djaglowski @schmikei

See Adding Labels via Comments if you do not have permissions to add labels yourself.

ammarasheikh · 2023-11-07T13:58:55Z

hi @djaglowski & @schmikei, can you please help?

schmikei · 2023-11-07T14:41:57Z

@ammarasheikh I apologize for the delay, my email has been swamped as of late!

Just confiriming, have you configured audit on the mongodbatlas side?

https://www.mongodb.com/docs/atlas/database-auditing/

and have the procedure steps been followed on that page?

ammarasheikh · 2023-11-07T15:11:12Z

hi @schmikei, Thanks for getting back. Yes I had enabled audit logs on the mongdodb atlas cluster.

schmikei · 2023-11-07T15:24:09Z

hi @schmikei, Thanks for getting back. Yes I had enabled audit logs on the mongdodb atlas cluster.

🤔 just confirming then are you able to use the atlas CLI to grab the audit logs you're looking for? And if you want to share more cluster info that may be helpful in trying to assist you!

https://www.mongodb.com/docs/atlas/cli/stable/command/atlas-logs-download/#atlas-logs-download

ammarasheikh · 2023-11-07T19:00:02Z

@schmikei , I was able to download the logs using Atlas CLI earlier but today I am getting an error "undefined response type". I am trying to resolve that issue. Just to provide additional details for the cluster it is a 3-node replica set, please see below screenshot. Another interesting thing which I see in the activity logs is logs continue to download successfully but I am not able to see them. The only message I see is the warning.

ammarasheikh · 2023-11-07T19:56:22Z

I am able to download the logs using atlas cli:

schmikei · 2023-11-07T21:00:17Z

Is it possible that some of your replica hosts (maybe some secondary) have 0 byte length for the interval of logs(I believe 5 minute windows from start of collection with a sliding window)? I.e. we downloaded a gzipped response of 0 bytes within the receiver? Are you not getting any process or audit logs from any host?

I think I was able to replicate the logged behavior using a low traffic cluster but primarily was getting these for the secondary hosts of the replica set.

I do think some improvement could be a logging enhancement to list the host and time range that the warn logging message is getting generated from.

ammarasheikh · 2023-11-07T21:48:22Z

Hi @schmikei, I have tried downloading the logs from secondary hosts and none of them has 0 byte . I know the cluster I am using is low traffic but still I can see logs generated from each replica node. Does the MongoDB Atlas receiver retrieve the logs from each replica host or just the primary as in the config we only provide project name whereas in the atlas cli we provide the individual host name to download the .gz logs?

schmikei · 2023-11-07T22:08:04Z

I have tried downloading the logs from secondary hosts and none of them has 0 byte

This depends mostly on the window the logs are requested, the default for atlas logs download goes back 1 days worth of logs whereas the receiver chunks its processing looking at the last 5 minutes of collection to keep the window of logs to process efficient I think is the intention.

https://www.mongodb.com/docs/atlas/cli/v1.9/command/atlas-logs-download/

Thats why I think at minimal we could improve the log that is emitted to print out the date range that is being requested from each host. I'm sort of strapped for time at the moment with some other projects but should be a small change if anybody wants to take a stab at it.

Does the MongoDB Atlas receiver retrieve the logs from each replica host or just the primary as in the config we only provide project name whereas in the atlas cli we provide the individual host name to download the .gz logs?

So right now audit logs are collected on a per-project basis by looking at each cluster per project scope and collects all hosts listed under that cluster scope. So that means each member of your replica set is being collected from for these logs when the receiver requests them.

I don't think I was involved on that initial decision, so I can't add context into why that is the current direction but is just the current state of the receiver and changing it might be considered a breaking change for some users.

ammarasheikh · 2023-11-08T17:19:30Z

Hi @schmikei, I am still not getting anywhere, I have added multiple projects to my config but still I am getting the same warning. I should at least see some logs as I know some of the clusters have activity so receiver should be able to retrieve logs. Is there a debugging option that could at least show which project the receiver has tried downloading the logs? Here's my new config with multiple projects, does it look good to you?

receivers:
mongodbatlas:
# existing metrics settings
public_key: xxxxxxx
private_key: xxxxxxxxxxxxxxxxxxxxxxxxxxx
granularity: PT1M
retry_on_failure:
enabled: true
#new logging settings
logs:
enabled: true #default true
projects: # List of projects to monitor logs from
- name: TRIPX_NONPROD # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: Hoth # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: Rahul_Mongo_db # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: aks-yoda-app-scheduler # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: alert-poc-prj # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
- name: dependency-leanix # Human readable name of the project
include_clusters: [] # optional list of clusters in a project to include for log collection, default empty
exclude_clusters: [] # option list of clusters in a project to exclude from log collection, default empty
collect_audit_logs: true
collect_host_logs: true # The user needs to enable this in Atlas so it needs to be enabled here
access_logs:
enabled: true
auth_result: false
page_size: 20000
max_pages: 10
poll_interval: 5m
exporters:
logging:
file:
path: /usr/src/app/log/mongologs.json
rotation:
max_megabytes: 10
max_days: 3
max_backups: 3
localtime: true
service:
pipelines:
logs:
receivers: [mongodbatlas]
exporters: [file]

schmikei · 2023-11-09T15:14:14Z

I'm not sure if it helps but here's a simple config that's been working for me

receivers: 
  mongodbatlas:
    public_key: <redacted>
    private_key: <redacted>
    resource_attributes:
      mongodb_atlas.cluster.name:
        enabled: true
    logs:
      enabled: true
      projects:
        - name: 'Project X'
          access_logs:
            enabled: true
            poll_interval: 5m
          collect_audit_logs: true


exporters:
  logging:
  file:
    path: logs.json

service:
  pipelines:
    metrics:
      receivers:
      - mongodbatlas
      exporters:
      - logging
    logs:
      receivers:
      - mongodbatlas
      exporters:
      - logging
      - file

Sample Log

{
              "timeUnixNano": "1699468210171000000",
              "observedTimeUnixNano": "1699468280813030000",
              "severityNumber": 9,
              "severityText": "INFO",
              "body": {
                "stringValue": "{ \"atype\" : \"clientMetadata\", \"ts\" : { \"$date\" : \"2023-11-08T18:30:10.171+00:00\" }, \"uuid\" : { \"$binary\" : \"PxJUeVRoRv2xnj/aPES5bQ==\", \"$type\" : \"04\" }, \"local\" : { \"ip\" : \"192.168.240.4\", \"port\" : 27017 }, \"remote\" : { \"ip\" : \"192.168.240.4\", \"port\" : 35968 }, \"users\" : [], \"roles\" : [], \"param\" : { \"localEndpoint\" : { \"ip\" : \"192.168.240.4\", \"port\" : 27017 }, \"clientMetadata\" : { \"application\" : { \"name\" : \"MongoDB CPS Module v13.6.3.8492 (git: 0ea489616c9470abf815c4eea7477f72f483a87a)\" }, \"driver\" : { \"name\" : \"mongo-go-driver\", \"version\" : \"v1.12.0-cloud\" }, \"os\" : { \"type\" : \"linux\", \"architecture\" : \"amd64\" }, \"platform\" : \"go1.20.10\" } }, \"result\" : 0 }"
              },
              "attributes": [
                {
                  "key": "atype",
                  "value": {
                    "stringValue": "clientMetadata"
                  }
                },
                {
                  "key": "local.ip",
                  "value": {
                    "stringValue": "192.168.240.4"
                  }
                },
                {
                  "key": "local.port",
                  "value": {
                    "intValue": "27017"
                  }
                },
                {
                  "key": "remote.ip",
                  "value": {
                    "stringValue": "192.168.240.4"
                  }
                },
                {
                  "key": "remote.port",
                  "value": {
                    "intValue": "35968"
                  }
                },
                {
                  "key": "uuid.binary",
                  "value": {
                    "stringValue": "PxJUeVRoRv2xnj/aPES5bQ=="
                  }
                },
                {
                  "key": "uuid.type",
                  "value": {
                    "stringValue": "04"
                  }
                },
                {
                  "key": "result",
                  "value": {
                    "intValue": "0"
                  }
                },
                {
                  "key": "param",
                  "value": {
                    "kvlistValue": {
                      "values": [
                        {
                          "key": "localEndpoint",
                          "value": {
                            "kvlistValue": {
                              "values": [
                                {
                                  "key": "ip",
                                  "value": {
                                    "stringValue": "192.168.240.4"
                                  }
                                },
                                {
                                  "key": "port",
                                  "value": {
                                    "doubleValue": 27017
                                  }
                                }
                              ]
                            }
                          }
                        },
                        {
                          "key": "clientMetadata",
                          "value": {
                            "kvlistValue": {
                              "values": [
                                {
                                  "key": "application",
                                  "value": {
                                    "kvlistValue": {
                                      "values": [
                                        {
                                          "key": "name",
                                          "value": {
                                            "stringValue": "MongoDB CPS Module v13.6.3.8492 (git: 0ea489616c9470abf815c4eea7477f72f483a87a)"
                                          }
                                        }
                                      ]
                                    }
                                  }
                                },
                                {
                                  "key": "driver",
                                  "value": {
                                    "kvlistValue": {
                                      "values": [
                                        {
                                          "key": "name",
                                          "value": {
                                            "stringValue": "mongo-go-driver"
                                          }
                                        },
                                        {
                                          "key": "version",
                                          "value": {
                                            "stringValue": "v1.12.0-cloud"
                                          }
                                        }
                                      ]
                                    }
                                  }
                                },
                                {
                                  "key": "os",
                                  "value": {
                                    "kvlistValue": {
                                      "values": [
                                        {
                                          "key": "type",
                                          "value": {
                                            "stringValue": "linux"
                                          }
                                        },
                                        {
                                          "key": "architecture",
                                          "value": {
                                            "stringValue": "amd64"
                                          }
                                        }
                                      ]
                                    }
                                  }
                                },
                                {
                                  "key": "platform",
                                  "value": {
                                    "stringValue": "go1.20.10"
                                  }
                                }
                              ]
                            }
                          }
                        }
                      ]
                    }
                  }
                },
                {
                  "key": "users",
                  "value": {
                    "arrayValue": {}
                  }
                },
                {
                  "key": "roles",
                  "value": {
                    "arrayValue": {}
                  }
                },
                {
                  "key": "log_name",
                  "value": {
                    "stringValue": "mongodb-audit-log.gz"
                  }
                }
              ],
              "traceId": "",
              "spanId": ""
            }

The only differences I could discern if all steps in enabling audit were followed is that we're only using an M10 cluster and running MongoDB 6.0 under the hood. Mongo 6 =>7 didn't change the log format afaik.

I've created a PR for the logging enhancement so hopefully that'll give you more information on your issue in the future!

**Description:** Enhances current logs by adding more context to the warning message and adds a debug log that will help orient users if they cannot reach that block of code. **Link to tracking Issue:** Resolves #28851 **Testing:** ```log 2023-11-09T10:11:17.716-0500 warn [email protected]/logs.go:245 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos.gz", "hostname": "cluster1-shard-00-00.t5hdg.mongodb.net", "startTime": "2023-11-09T10:06:16.618-0500", "endTime": "2023-11-09T10:11:16.618-0500"} 2023-11-09T10:11:17.946-0500 info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 28} 2023-11-09T10:11:18.131-0500 warn [email protected]/logs.go:274 Attempted to retrieve audit logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "hostname": "cluster1-shard-00-00.t5hdg.mongodb.net", "log": "mongos-audit-log.gz", "startTime": "2023-11-09T10:06:16.618-0500", "endTime": "2023-11-09T10:11:16.618-0500"} ``` **Documentation:** <Describe the documentation added.>

ammarasheikh · 2023-11-15T20:57:01Z

@schmikei, thanks for adding the enhancement. Unfortunately I am not able to see the logging enhancement. I am still seeing the same warning messages and not seeing the enhancement to list the host and time range that the warning logging message is getting generated from.

2023-11-15T20:44:32.304Z info [email protected]/telemetry.go:84 Setting up own telemetry... ││ 2023-11-15T20:44:32.304Z info [email protected]/telemetry.go:201 Serving Prometheus metrics {"address": ":8888", "level": "Basic"} ││ 2023-11-15T20:44:32.304Z info [email protected]/exporter.go:275 Deprecated component. Will be removed in future releases. {"kind": "exporter", "data_type": "logs", "name": "logging"} ││ 2023-11-15T20:44:32.304Z info [email protected]/exporter.go:275 Deprecated component. Will be removed in future releases. {"kind": "exporter", "data_type": "metrics", "name": "logging"} ││ 2023-11-15T20:44:32.306Z info [email protected]/service.go:143 Starting otelcol-contrib... {"Version": "0.88.0", "NumCPU": 4} ││ 2023-11-15T20:44:32.306Z info extensions/extensions.go:33 Starting extensions... ││ 2023-11-15T20:44:32.306Z info [email protected]/service.go:169 Everything is ready. Begin running and processing data. ││ 2023-11-15T20:44:32.840Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 146} ││ 2023-11-15T20:44:32.980Z warn [email protected]/logs.go:229 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos.gz"} ││ 2023-11-15T20:44:33.133Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 54} ││ 2023-11-15T20:44:33.268Z warn [email protected]/logs.go:259 Attempted to retrieve audit logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos-aud ││ 2023-11-15T20:44:33.553Z warn [email protected]/logs.go:229 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos.gz"} ││ 2023-11-15T20:44:33.831Z warn [email protected]/logs.go:259 Attempted to retrieve audit logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos-aud ││ 2023-11-15T20:44:33.963Z warn [email protected]/logs.go:229 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongodb.gz" ││ 2023-11-15T20:44:34.079Z warn [email protected]/logs.go:229 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos.gz"} ││ 2023-11-15T20:44:34.181Z warn [email protected]/logs.go:259 Attempted to retrieve audit logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongodb-au ││ 2023-11-15T20:44:34.306Z warn [email protected]/logs.go:259 Attempted to retrieve audit logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos-aud ││ 2023-11-15T20:44:49.517Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 20, "metrics": 420, "data points": 3116} ││ 2023-11-15T20:47:50.884Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 30, "metrics": 439, "data points": 3212} ││ 2023-11-15T20:49:32.644Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1} ││ 2023-11-15T20:49:34.783Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 151} ││ 2023-11-15T20:49:34.914Z warn [email protected]/logs.go:229 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos.gz"} ││ 2023-11-15T20:49:35.055Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 50}

schmikei · 2023-11-15T21:01:45Z

@ammarasheikh the change won't be in until v0.89.0 is my understanding, when that gets released feel free to try it out!

ammarasheikh · 2023-11-15T21:30:38Z

ok sounds good, thanks

…telemetry#29084) **Description:** Enhances current logs by adding more context to the warning message and adds a debug log that will help orient users if they cannot reach that block of code. **Link to tracking Issue:** Resolves open-telemetry#28851 **Testing:** ```log 2023-11-09T10:11:17.716-0500 warn [email protected]/logs.go:245 Attempted to retrieve host logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "log": "mongos.gz", "hostname": "cluster1-shard-00-00.t5hdg.mongodb.net", "startTime": "2023-11-09T10:06:16.618-0500", "endTime": "2023-11-09T10:11:16.618-0500"} 2023-11-09T10:11:17.946-0500 info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 28} 2023-11-09T10:11:18.131-0500 warn [email protected]/logs.go:274 Attempted to retrieve audit logs but received 0 logs {"kind": "receiver", "name": "mongodbatlas", "data_type": "logs", "error": "EOF", "hostname": "cluster1-shard-00-00.t5hdg.mongodb.net", "log": "mongos-audit-log.gz", "startTime": "2023-11-09T10:06:16.618-0500", "endTime": "2023-11-09T10:11:16.618-0500"} ``` **Documentation:** <Describe the documentation added.>

CJ619 · 2024-06-04T23:26:38Z

@ammarasheikh were you ever able to resolve this issue?

ammarasheikh added the needs triage New item requiring triage label Nov 1, 2023

github-actions bot added the receiver/mongodbatlas label Nov 1, 2023

github-actions bot mentioned this issue Nov 7, 2023

Weekly Report: 2023-10-31 - 2023-11-07 #29000

Closed

schmikei mentioned this issue Nov 9, 2023

[receiver/mongodbatlas] Logging enhancement for logs retrieval #29084

Merged

crobert-1 added question Further information is requested and removed needs triage New item requiring triage labels Nov 10, 2023

djaglowski closed this as completed in #29084 Nov 14, 2023

crobert-1 mentioned this issue Mar 26, 2024

Mongodb Atlas receiver not able to pull logs issue #31978

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mongodb Atlas receiver not able to pull logs issue #28851

Mongodb Atlas receiver not able to pull logs issue #28851

ammarasheikh commented Nov 1, 2023

github-actions bot commented Nov 1, 2023

ammarasheikh commented Nov 7, 2023

schmikei commented Nov 7, 2023 •

edited

Loading

ammarasheikh commented Nov 7, 2023

schmikei commented Nov 7, 2023

ammarasheikh commented Nov 7, 2023

ammarasheikh commented Nov 7, 2023

schmikei commented Nov 7, 2023

ammarasheikh commented Nov 7, 2023

schmikei commented Nov 7, 2023

ammarasheikh commented Nov 8, 2023

schmikei commented Nov 9, 2023

ammarasheikh commented Nov 15, 2023

schmikei commented Nov 15, 2023

ammarasheikh commented Nov 15, 2023

CJ619 commented Jun 4, 2024

Mongodb Atlas receiver not able to pull logs issue #28851

Mongodb Atlas receiver not able to pull logs issue #28851

Comments

ammarasheikh commented Nov 1, 2023

Component(s)

Describe the issue you're reporting

github-actions bot commented Nov 1, 2023

ammarasheikh commented Nov 7, 2023

schmikei commented Nov 7, 2023 • edited Loading

ammarasheikh commented Nov 7, 2023

schmikei commented Nov 7, 2023

ammarasheikh commented Nov 7, 2023

ammarasheikh commented Nov 7, 2023

schmikei commented Nov 7, 2023

ammarasheikh commented Nov 7, 2023

schmikei commented Nov 7, 2023

ammarasheikh commented Nov 8, 2023

schmikei commented Nov 9, 2023

ammarasheikh commented Nov 15, 2023

schmikei commented Nov 15, 2023

ammarasheikh commented Nov 15, 2023

CJ619 commented Jun 4, 2024

schmikei commented Nov 7, 2023 •

edited

Loading