Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved Datadog integration #385

Merged
merged 3 commits into from
Feb 15, 2021
Merged

Conversation

markbastiaans
Copy link
Contributor

@markbastiaans markbastiaans commented Dec 14, 2020

This PR includes the following changes to the Datadog and Telegraf integrations:

  • Use Telegraf agent for collecting PostgreSQL metrics and database diskstorage size metric. Telegraf will send these metrics directly to Datadog if Datadog is enabled.
  • Use Datadog IoT and trace agents for collecting application metrics, JMX metrics, logs and traces
  • Use official Datadog Cloud Foundry buildpack. This saves 400+ MB of application disk space and makes the integration more maintainable in the future.
  • Added logs redaction - email addresses will be redacted by default. Can be switched off by setting the DATADOG_LOGS_REDACTION environment variable to false.
  • Removed direct Databroker dependencies in Datadog module
  • Refactoring and improvements: tags, configuration templates and more

Known Limitations

Telegraf does not support (Datadog) metric types correctly yet (e.g. rate, counter, gauge). This means that all database metrics are currently pushed to Datadog as a gauge.

The most important metrics (before_xid_wraparound, connections, database_size, db.count, locks, max_connections, percent_usage_connections, table.count, deadlocks) are gauges and are compatible with the Datadog PostgreSQL integration and associated dashboards.

If you do require the additional rate and counter metrics, there is a workaround available. First, set the DATADOG_DATABASE_RATE_COUNT_METRICS environment variable to true. After that variable is enabled, the rate and counter metrics are suffixed by either _count or _rate to prevent collisions with the official Datadog metrics. In the Datadog UI, the metric type and unit can be changed to reflect this. We also set a helpful interval tag (10s) which can be used here. Additionally, gauge metrics can be rolled up in Datadog dashboards. The correct type and unit for other submitted metrics can be found here.

@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch 5 times, most recently from a7c61e3 to 883498b Compare December 18, 2020 15:51
@abhishekamte-mx abhishekamte-mx force-pushed the datadog_integration_improvements branch from b3ec70a to 2415a96 Compare December 21, 2020 08:04
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch 3 times, most recently from a8b9911 to 5431cfb Compare January 7, 2021 10:40
@markbastiaans markbastiaans requested a review from cwesdorp January 7, 2021 11:12
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch 4 times, most recently from 5add5e6 to 3b4f9d1 Compare January 13, 2021 13:04
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch 3 times, most recently from b29e1c1 to 60b504a Compare January 14, 2021 14:59
@markbastiaans markbastiaans requested review from matyaskuti and serg-firsov and removed request for cwesdorp January 14, 2021 15:02
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch 3 times, most recently from 5ac2251 to 2ea7f47 Compare January 15, 2021 13:44
@markbastiaans markbastiaans marked this pull request as ready for review January 15, 2021 13:46
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch 2 times, most recently from 02165d8 to cfd8cb5 Compare January 28, 2021 11:46
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch 3 times, most recently from db4f75d to 420c6db Compare January 28, 2021 14:08
@markbastiaans markbastiaans removed the request for review from serg-firsov January 28, 2021 14:46
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch 4 times, most recently from d283e0a to 1c9e101 Compare February 11, 2021 13:34
Copy link
Contributor

@matyaskuti matyaskuti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay in general, a few small comments and questions, nothing big related to the overall goal.

buildpack/databroker/__init__.py Show resolved Hide resolved
buildpack/databroker/__init__.py Show resolved Hide resolved
tests/unit/test_datadog.py Outdated Show resolved Hide resolved
buildpack/datadog.py Show resolved Hide resolved
@markbastiaans markbastiaans removed the request for review from udh-mendix February 15, 2021 09:29
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch from 9e7da3d to 4c25af7 Compare February 15, 2021 10:22
@markbastiaans markbastiaans force-pushed the datadog_integration_improvements branch from 4c25af7 to 7596b30 Compare February 15, 2021 10:36
@markbastiaans markbastiaans merged commit 72f1270 into develop Feb 15, 2021
@markbastiaans markbastiaans deleted the datadog_integration_improvements branch February 15, 2021 10:52
markbastiaans added a commit that referenced this pull request Feb 22, 2021
* Improved Datadog integration
* Move loggregator util function to metrics module
* Update Python dependencies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants