Collect mysql statement samples & execution plans

[djova]

What does this PR do?

Adds a new feature to "Deep Database Monitoring", enabling collection of statement samples and execution plans.

Follow-up to:

• Support MySQL statement-level metrics for deep database monitoring #7851 - mysql DBM statement metrics
• Collect postgres statement samples & execution plans for deep database monitoring #8627 - Collect postgres statement samples & execution plans

How it works

If enabled, a python thread is launched during a regular check run:

• collects statement samples at the configured rate limit (default 1 collection per second)
• maintains its own pymysql connection as pymysql is not thread safe so we can't share with the main check
• collects execution plans through a mysql procedure that the user must install into each database being monitored (if we wanted the agent to collect execution plans directly by running EXPLAIN then it would need full write permission to all tables)
• shuts down if it detects that the main check has stopped running

During one "collection" we do the following:

1. read out all new statements from performance_schema.events_statement_{current|history|history_long}
2. try to collect an execution plan for each statement
3. submit events directly to the new database monitoring event intake

Rate Limiting

There are several different rate limits to keep load on the database to a minimum and to avoid reingesting duplicate events:

• collections_per_second: limits how often collections are done (each collection is a query to an events_statements_* table)
• explained_statements_cache: ttl limits how often we attempt to collect an execution plan for a given normalized query
• seen_samples_cache: ttl limits how often we ingest statement samples for the same normalized query and execution plan

Events statements tables

The check will collect samples from the best available events_statements table. It's up to the user to enable the required events_statements consumers. Some managed databases like RDS Aurora replicas don't support events_statements_history_long in which case we fall back to one of the other tables.

1. events_statements_history_long - preferred as it has the longest retention giving us the highest chance of catching infrequent & fast queries
2. events_statements_current - less likely to catch infrequent & fast queries
3. events_statements_history - least preferred table because

Configuration

statement_samples:
  enabled: false
  # default rate depends on which events_statements table is being used. user can override.
  collections_per_second: -1
  # the best table is chosen automatically by the check. user can override.
  events_statements_table: ''
  events_statements_row_limit: 5000
  explain_procedure: 'explain_statement'
  fully_qualified_explain_procedure: 'datadog.explain_statement'
  events_statements_temp_table_name: 'datadog.temp_events'
  events_statements_enable_procedure: 'datadog.enable_events_statements_consumers'

Motivation

Collect statement samples & execution plans, enabling deeper insight into what's running on the database and how queries are being executed.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

[jtappa]

Some suggestions to switch to a more active voice and consistency across descriptions.

[djova]

@olivielpeau @hush-hush I added support for cancel here. I think we should still keep the "stop thread if it notices the check is no longer running" logic as a fallback if the main check stops running for any other reason (agent overload, or a bug somewhere where it gets stuck).

[olivielpeau]

ok, LGTM.

I don't have a strong opinion about the fallback. Just a couple of thoughts: if the check is stuck it'll cause other problems (it'll hog a check runner, etc), but the fallback would at least free up some resources that's true. If the check is inactive because of an agent overload, and the fallback mechanism stops the background thread, you'll want to make sure the background thread can be started up again cleanly when the check is run again.

[djova]

you'll want to make sure the background thread can be started up again cleanly when the check is run again.

Yes that is the current behavior. On every check run the check will start up the thread if it is not currently running.

[djova]

Some suggestions to switch to a more active voice and consistency across descriptions.

Thanks @jtappa for the review!

[jtappa]

👍🏻 from docs!