[receiver/mongodbatlas] Alerts Retrieval

[schmikei]

Description: Spikes out some retrieval of alerts. This is because sometimes firewalls block mongodbatlas from forwarding to the collector.

Example JSON output:

    {
      "resource": {
        "attributes": [
          {
            "key": "mongodbatlas.group.id",
            "value": {
              "stringValue": "5bc762b579358e3332046e6a"
            }
          },
          {
            "key": "mongodbatlas.alert.config.id",
            "value": {
              "stringValue": "6331df916966412533ce3eaa"
            }
          },
          {
            "key": "mongodbatlas.cluster.name",
            "value": {
              "stringValue": "Cluster0"
            }
          },
          {
            "key": "mongodbatlas.replica_set.name",
            "value": {
              "stringValue": "atlas-zx8u63-shard-0"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1664304327000000000",
              "observedTimeUnixNano": "1664305434847319000",
              "severityNumber": "SEVERITY_NUMBER_WARN",
              "body": {
                "stringValue": "{\"id\":\"633335c99998645b1803c60b\",\"groupId\":\"5bc762b579358e3332046e6a\",\"alertConfigId\":\"6331df916966412533ce3eaa\",\"eventTypeName\":\"OUTSIDE_METRIC_THRESHOLD\",\"created\":\"2022-09-27T17:41:29Z\",\"updated\":\"2022-09-27T18:45:27Z\",\"status\":\"OPEN\",\"lastNotified\":\"2022-09-27T18:45:35Z\",\"hostnameAndPort\":\"atlas-zx8u63-shard-00-02.t5hdg.mongodb.net:27017\",\"metricName\":\"DB_STORAGE_TOTAL\",\"currentValue\":{\"number\":9080832,\"units\":\"BYTES\"},\"replicaSetName\":\"atlas-zx8u63-shard-0\",\"clusterName\":\"Cluster0\"}"
              },
              "attributes": [
                {
                  "key": "event.domain",
                  "value": {
                    "stringValue": "mongodbatlas"
                  }
                },
                {
                  "key": "event.name",
                  "value": {
                    "stringValue": "OUTSIDE_METRIC_THRESHOLD"
                  }
                },
                {
                  "key": "status",
                  "value": {
                    "stringValue": "OPEN"
                  }
                },
                {
                  "key": "created",
                  "value": {
                    "stringValue": "2022-09-27T17:41:29Z"
                  }
                },
                {
                  "key": "updated",
                  "value": {
                    "stringValue": "2022-09-27T18:45:27Z"
                  }
                },
                {
                  "key": "id",
                  "value": {
                    "stringValue": "633335c99998645b1803c60b"
                  }
                },
                {
                  "key": "metric.name",
                  "value": {
                    "stringValue": "DB_STORAGE_TOTAL"
                  }
                },
                {
                  "key": "type_name",
                  "value": {
                    "stringValue": "OUTSIDE_METRIC_THRESHOLD"
                  }
                },
                {
                  "key": "last_notified",
                  "value": {
                    "stringValue": "2022-09-27T18:45:35Z"
                  }
                },
                {
                  "key": "resolved",
                  "value": {
                    "stringValue": ""
                  }
                },
                {
                  "key": "acknowledgement.comment",
                  "value": {
                    "stringValue": ""
                  }
                },
                {
                  "key": "acknowledgement.username",
                  "value": {
                    "stringValue": ""
                  }
                },
                {
                  "key": "acknowledgement.until",
                  "value": {
                    "stringValue": ""
                  }
                },
                {
                  "key": "metric.value",
                  "value": {
                    "doubleValue": 9080832
                  }
                },
                {
                  "key": "metric.units",
                  "value": {
                    "stringValue": "BYTES"
                  }
                },
                {
                  "key": "net.peer.name",
                  "value": {
                    "stringValue": "atlas-zx8u63-shard-00-02.t5hdg.mongodb.net"
                  }
                },
                {
                  "key": "net.peer.port",
                  "value": {
                    "intValue": "27017"
                  }
                }
              ],
              "traceId": "",
              "spanId": ""
            }
          ]
        }
      ]
    }

Implemented inside this is usage of the storage extension that can optionally specify a storage extension to persist checkpoints as the API does not support filtering between a time range.

Example config

receivers:
  mongodbatlas:
    public_key: redacted
    private_key: redacted
    alerts:
      enabled: true
      mode: retrieve
      projects:
      - name: Project 1
        include_clusters: [Cluster0]
      poll_interval: 5m
    storage: file_storage

extensions:
  file_storage:
    directory: /Users/keithschmitt/go-workspace/src/github.com/open-telemetry/opentelemetry-collector-contrib/tmp

Link to tracking Issue: Resolves #14121

Testing: Adds unit testing/psuedo integration testing

Documentation: More documentation on alerts has been added.

[djaglowski]

Lots of feedback, but I think there's also a general design issue that will lead to scalability problems.

Specifically, I don't think it's reasonable to assume that we can process all of the alerts that are available to us at a given time. We need to limit the size of the plog.Logs emitted by the receiver.

If we don't, we may be passing a payload down the pipeline that is so large that it is not easily processed or exported. We may even eat up all memory available to the collector.

This can be made scalable by emitting multiple times per poll cycle, when appropriate. This gives the pipeline an opportunity to process and export the data.

The easiest way to do this would be to emit a plog.Logs once per page. The sdk allows page size to be configured, so at a later time we may choose to expose that to users so that they have a way to further limit the size of payloads.

Implementing this solution will affect the way you're tracking the lastRecordedTime, so be careful to ensure it is only saved to the receiver once per poll cycle.

[djaglowski]

This looks good @schmikei. Thanks for iterating on it.