Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Collect mysql statement samples & execution plans (#8629)
* Collect mysql statement samples & execution plans **What does this PR do?** Adds a new feature to "Deep Database Monitoring", enabling collection of statement samples and execution plans. Follow-up to #7851. **How it works** If enabled, a python thread is launched during a regular check run: * collects statement samples at the configured rate limit (default 1 collection per second) * maintains its own `pymysql` connection as `pymysql` is not thread safe so we can't share with the main check * collects execution plans through a mysql procedure that the user must install into each database being monitored (if we wanted the agent to collect execution plans directly by running `EXPLAIN` then it would need full write permission to all tables) * shuts down if it detects that the main check has stopped running During one "collection" we do the following: 1. read out all new statements from `performance_schema.events_statement_{summary|history|history_long}` 1. try to collect an execution plan for each statement 1. submit events directly to the new database monitoring event intake **Rate Limiting** There are several different rate limits to keep load on the database to a minimum and to avoid reingesting duplicate events: * `collections_per_second`: limits how often collections are done (each collection is a query to an `events_statements_*` table) * `explained_statements_cache`: ttl limits how often we attempt to collect an execution plan for a given normalized query * `seen_samples_cache`: ttl limits how often we ingest statement samples for the same normalized query and execution plan **events statements tables** The check will collect samples from the best available `events_statements` table. It's up to the user to enable the required `events_statements` consumers. Some managed databases like RDS Aurora replicas don't support `events_statements_history_long` in which case we fall back to one of the other tables. 1. `events_statements_history_long` - preferred as it has the longest retention giving us the highest chance of catching infrequent & fast queries 1. `events_statements_current` - less likely to catch infrequent & fast queries 1. `events_statements_history` - least preferred table because **Configuration** ```yaml statement_samples: enabled: false # default rate depends on which events_statements table is being used. user can override. collections_per_second: -1 # the best table is chosen automatically by the check. user can override. events_statements_table: '' events_statements_row_limit: 5000 explain_procedure: 'explain_statement' fully_qualified_explain_procedure: 'datadog.explain_statement' events_statements_temp_table_name: 'datadog.temp_events' ``` **Motivation** Collect statement samples & execution plans, enabling deeper insight into what's running on the database and how queries are being executed. * style * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update spec * support cancel * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Update mysql/assets/configuration/spec.yaml Co-authored-by: Jorie Helwig <[email protected]> * Conf example update from spec * fix client race condition in tests * remove unnecessary * style * use json dep * bump base Co-authored-by: Jorie Helwig <[email protected]>
- Loading branch information