Add MySQL query metricset with lightweight module and SQL helper #18955
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sayden
added
enhancement
Metricbeat
sayden
changed the title
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
sayden
force-pushed
the
feature/mb/sql-helper
branch
from
e3661af to
45e85bd
Compare
sayden
force-pushed
the
feature/mb/sql-helper
branch
from
b14cf5f to
c34a9fd
Compare
|
I'm thinking to remove the Having it as a lightweight module, it is easy to include it later. |
|
Pinging @elastic/integrations-services (Team:Services) |
ChrsMark
reviewed
Jul 7, 2020
sayden
force-pushed
the
feature/mb/sql-helper
branch
from
876ab17 to
41941aa
Compare
jsoriano
previously requested changes
Jul 7, 2020
There was a problem hiding this comment.
This is looking good, I think that is a good idea to have generic metricsets on the specific modules to have documentation together and a more consistent experience for users using an only db.
Changelog entries will be needed, there are many things here we should mention:
- New query metricset in the mysql module.
- New performance metricset in the mysql module.
- Add support for booleans in the generic sql module.
- Null values are not reported in the generic sql module.
Maybe for the developer changelog:
- New sql helper for metricbeat modules.
|
Very good comments indeed. Thanks! Will take a look |
kaiyan-sheng
reviewed
Jul 7, 2020
kaiyan-sheng
reviewed
Jul 7, 2020
ChrsMark
reviewed
Jul 8, 2020
jsoriano
reviewed
Jul 9, 2020
bmorelli25
reviewed
Jul 13, 2020
metricbeat/docs/modules/sql.asciidoc
Outdated
| driver: "postgres" | ||
| sql_query: "SELECT * FROM pg_catalog.pg_tables pt WHERE schemaname ='pg_catalog'" | ||
| sql_response_format: table | ||
| --- |
There was a problem hiding this comment.
This should fix the doc build
Suggested change
| --- | |
| ---- |
There was a problem hiding this comment.
Oh but this is generated. I don't know where this is is coming from. @dedemorton probably will
There was a problem hiding this comment.
Oh please, I cannot believe it 😄
There was a problem hiding this comment.
Ok, pushed and fixed. Thank you very much 😃
sayden
added a commit
to sayden/beats
that referenced
this pull request
Jul 13, 2020
…stic#18955) (cherry picked from commit d0f7058)
Merged
5 tasks
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jul 14, 2020
* upstream/master: (25 commits) [Elastic Agent] Send checkin payload to Fleet (elastic#19857) [Ingest Manager] Fixed tests across agent elastic#19877 [Ingest Manager] Fix serialization test elastic#19876 Fix service start type mapping in windows/service metricset (elastic#19551) ci: Change comment trigger detection method (elastic#19827) Add 21 autogenerated filesets from rsa2elk devices (elastic#19713) [Ingest Manager] Agent config cleanup (elastic#19848) libbeat/publisher/pipeline: fix data races (elastic#19821) Update monitoring-internal-collection.asciidoc (elastic#19422) (elastic#19697) [Elastic Agent] Trust exchange endpoint must bind to 127.0.0.1 (elastic#19861) Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (elastic#19159) Add azure billing metricset (elastic#19207) Add support for appinsights in the metricbeat azure module (elastic#18940) Add MySQL query metricset with lightweight module and SQL helper (elastic#18955) [Ingest Manager] Refuse invalid stream values in configuration (elastic#19587) Do not use vendor during integration tests (elastic#19839) LIBBEAT: Enhancement Convert dissected values from String to other basic data types and IP (elastic#18683) [Elastic Agent] Remove support for "logs" and only support logfile (elastic#19761) [CI] support windows-2012 (elastic#19773) Do not update go.mod during packaging and testing (elastic#19823) ...
This was referenced Jul 22, 2020
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds a SQL helper (at Metricbeat level) with logic to execute SQL queries and return their results. The module can use this abstraction layer to simply pass in the SQL driver and the queries they need to execute to later create Beats events.
This PR introduce:
sqlmodule.sqlmodule.queryMetricset in MySQL module which uses the SQL helper. This Metricset is to use it as a Lightweight "parent" module.performancemetricset using thequerymetricset as Lightweight parent module. Use it as a guidance about how to make it work.overviewmetricset with a bunch of key metrics, also using Lightweight modules. This is the metricset to remove of this PR.sqlmodule was moved to thesqlhelper.Why is it important?
The way the
sqlwas working was a bit limited compared to the underlying power it has. Moving it one layer up from the modules, allows the use of its logic by any module while maintaining each module functionality regarding specific DSN or security, for example, instead of introducing all that logic into thesqlmodule.The way it works now also allows to create metricsets on CockroachDB module using lightweight modules.
Checklist
[ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
Related issues
Fixes: #18898
Implements: #15048