Cherry-pick #17156 to 7.x: [BUG] Fix "regular expression has redundant nested repeat operator" in MySQL slowlog module.

[adriansr]

Cherry-pick of PR #17156 to 7.x branch. Original message:

What does this PR do?

Remove a superfluous wildcard from EXPLAIN pattern used in the the grok pattern in MySQL slowlog ingest pipeline.

Why is it important?

It will fix excessive log messages produced every time the pipeline is used:

Mar 18 18:12:07 elk elasticsearch[29353]: regular expression has redundant nested repeat operator * /^# User@Host: (?<USER:user.name>(?:[a-zA-Z0-9._-]+))(\[(?<USER:mysql.slowlog.current_user>(?:[a-zA-Z0-9._-]+))\])? @ (?<HOSTNAME:source.domain>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))? \[(?<IP:source.ip>(?:(?:((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?)|(?:(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]))))?\](?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*))(Id:(?:\s*)(?<NUMBER:mysql.thread_id:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Thread_id:(?:\s*)(?<NUMBER:mysql.thread_id>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Schema:(?:\s*)(?<WORD:mysql.slowlog.schema>\b\w+\b)?(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Last_errno: (?<NUMBER:mysql.slowlog.last_errno:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Killed: (?<NUMBER:mysql.slowlog.killed:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(QC_hit: (?<WORD:mysql.slowlog.query_cache_hit>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Query_time: (?<NUMBER:temp.duration:float>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Lock_time: (?<NUMBER:mysql.slowlog.lock_time.sec:float>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Rows_sent: (?<NUMBER:mysql.slowlog.rows_sent:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Rows_examined: (?<NUMBER:mysql.slowlog.rows_examined:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Rows_affected: (?<NUMBER:mysql.slowlog.rows_affected:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Thread_id: (?<NUMBER:mysql.thread_id>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Errno: (?<NUMBER:mysql.slowlog.last_errno:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Killed: (?<NUMBER:mysql.slowlog.killed:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Bytes_received: (?<NUMBER:mysql.slowlog.bytes_received:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Bytes_sent: (?<NUMBER:mysql.slowlog.bytes_sent:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Read_first: (?<NUMBER:mysql.slowlog.read_first:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Read_last: (?<NUMBER:mysql.slowlog.read_last:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Read_key: (?<NUMBER:mysql.slowlog.read_key:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Read_next: (?<NUMBER:mysql.slowlog.read_next:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Read_prev: (?<NUMBER:mysql.slowlog.read_prev:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Read_rnd: (?<NUMBER:mysql.slowlog.read_rnd:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Read_rnd_next: (?<NUMBER:mysql.slowlog.read_rnd_next:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Sort_merge_passes: (?<NUMBER:mysql.slowlog.sort_merge_passes:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Sort_range_count: (?<NUMBER:mysql.slowlog.sort_range_count:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Sort_rows: (?<NUMBER:mysql.slowlog.sort_rows:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Sort_scan_count: (?<NUMBER:mysql.slowlog.sort_scan_count:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Created_tmp_disk_tables: (?<NUMBER:mysql.slowlog.tmp_disk_tables:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Created_tmp_tables: (?<NUMBER:mysql.slowlog.tmp_tables:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Tmp_tables: (?<NUMBER:mysql.slowlog.tmp_tables:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Tmp_disk_tables: (?<NUMBER:mysql.slowlog.tmp_disk_tables>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Tmp_table_sizes: (?<NUMBER:mysql.slowlog.tmp_table_sizes:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Start: (?<TIMESTAMP_ISO8601:event.start>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(End: (?<TIMESTAMP_ISO8601:event.end>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01][0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(InnoDB_trx_id: (?<WORD:mysql.slowlog.innodb.trx_id>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(QC_Hit: (?<WORD:mysql.slowlog.query_cache_hit>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Full_scan: (?<WORD:mysql.slowlog.full_scan>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Full_join: (?<WORD:mysql.slowlog.full_join>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Tmp_table: (?<WORD:mysql.slowlog.tmp_table>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Tmp_table_on_disk: (?<WORD:mysql.slowlog.tmp_table_on_disk>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Filesort: (?<WORD:mysql.slowlog.filesort>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Filesort_on_disk: (?<WORD:mysql.slowlog.filesort_on_disk>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Merge_passes: (?<NUMBER:mysql.slowlog.merge_passes:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Priority_queue: (?<WORD:mysql.slowlog.priority_queue>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(No InnoDB statistics available for this query(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(InnoDB_IO_r_ops: (?<NUMBER:mysql.slowlog.innodb.io_r_ops:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(InnoDB_IO_r_bytes: (?<NUMBER:mysql.slowlog.innodb.io_r_bytes:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(InnoDB_IO_r_wait: (?<NUMBER:mysql.slowlog.innodb.io_r_wait.sec:float>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(InnoDB_rec_lock_wait: (?<NUMBER:mysql.slowlog.innodb.rec_lock_wait.sec:float>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(InnoDB_queue_wait: (?<NUMBER:mysql.slowlog.innodb.queue_wait.sec:float>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(InnoDB_pages_distinct: (?<NUMBER:mysql.slowlog.innodb.pages_distinct:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Log_slow_rate_type: (?<WORD:mysql.slowlog.log_slow_rate_type>\b\w+\b)(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(Log_slow_rate_limit: (?<NUMBER:mysql.slowlog.log_slow_rate_limit:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:([ #
Mar 18 18:12:07 elk elasticsearch[29353]: ]*)))?(?:(# explain:.*
Mar 18 18:12:07 elk elasticsearch[29353]: |#\s*
Mar 18 18:12:07 elk elasticsearch[29353]: )*)?(use (?<WORD:mysql.slowlog.schema>\b\w+\b);
Mar 18 18:12:07 elk elasticsearch[29353]: )?SET timestamp=(?<NUMBER:mysql.slowlog.timestamp:long>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))));
Mar 18 18:12:07 elk elasticsearch[29353]: (?<GREEDYMULTILINE:mysql.slowlog.query>(.|
Mar 18 18:12:07 elk elasticsearch[29353]: )*)/

Checklist

• My code follows the style guidelines of this project
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works

Related issues

• Closes Filebeat mysql slowlog module regular expression has redundant nested repeat operator. #17086