Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch #26737

adriansr · 2021-07-06T13:42:38Z

What does this PR do?

This PR makes two changes:

Improve the pipeline/compatibility code so that all processors in a pipeline are scanned and acted-upon to ensure compatibility. This means:
- Scan processors in on_failure section (both the pipeline's and each individual processor on-failure handler).
- Scan the inner processor in a foreach processor.
Add a new CI stage, module-compat-7.11, to filebeat/Jenkinsfile.yml and x-pack/filebeat/Jenkinsfile.yml, in order to run Filebeat modules tests under ES 7.11.2 to ensure that all pipelines are functional.

This test uses a new flag, TESTING_FILEBEAT_SKIP_DIFF, to instruct the integration test to skip the comparison between the generated documents and the golden/expected files. The test will ensure that the pipeline loads, there are no ingest errors, the fields in the generated documents are valid and the number of returned documents matches the expected. This is intended to avoid having to maintain multiple versions of the golden files due to differences between ES versions and available processors.

Why is it important?

To ensure that Filebeat is backwards-compatible with 7.x releases of ES.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~[ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

Related issues

[Filebeat] Module incompatibility with older ES/Kibana versions #26629

elasticmachine · 2021-07-06T13:47:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-07-07T08:49:51.021+0000
Duration: 136 min 33 sec
Commit: 956de5d

Test stats 🧪

Test	Results
Failed	0
Passed	49191
Skipped	5395
Total	54586

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	49191
Skipped	5395
Total	54586

This reverts commit 9357409.

While testing those modules in 7.11-compat mode, the tests would fail because the pipelines leave some unexpected fields behind. This cleans them up. This is caused by the uri_parts processor being removed and leaving behind a temporary field that is cleaned up by the `remove_if_successful` flag.

elasticmachine · 2021-07-07T09:06:05Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh

LGTM. This will help a lot in catching future problems. 🥇

andrewkroh · 2021-07-07T12:58:27Z

filebeat/fileset/compatibility.go

 // adaptPipelineForCompatibility iterates over all processors in the pipeline
 // and adapts them for version of Elasticsearch used. Adapt can mean modifying
 // processor options or removing the processor.
 func adaptPipelineForCompatibility(esVersion common.Version, pipelineID string, content map[string]interface{}, log *logp.Logger) (err error) {
-	p, ok := content["processors"]
+	log = log.With("pipelineID", pipelineID)


I think lower snake case has been used almost everywhere else for key names passed to the logger. Recommend change for consistency.

elasticmachine · 2021-07-07T15:36:06Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-07-07T13:18:14.051+0000
Duration: 137 min 46 sec
Commit: 9ee4d68

Test stats 🧪

Test	Results
Failed	0
Passed	49191
Skipped	5395
Total	54586

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	49191
Skipped	5395
Total	54586

* upstream/master: (430 commits) CI: increase timeout (elastic#26764) Heartbeat: add datastream fields to synthetics (elastic#26774) Osquerybeat: Change the query timeout from 3 secs to 60 secs (elastic#26775) Remove experimental warning for inputs with variables. (elastic#26762) Add latest k8s versions in testing (elastic#26729) change type of max_bytes to ByteType (elastic#26699) [Elastic Agent] Fix broken enrollment command (elastic#26749) Update agent managed manifest to include enrolment token variable (elastic#26756) Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch (elastic#26737) Forward port changelog for 7.13.3 (elastic#26731) to master (elastic#26754) Upgrade PyYAML dependency used for tests (elastic#26746) Add agent fleet enrolment k8s manifest (elastic#26566) CI: retry the step only (elastic#26736) Osquerybeat: Fix the configuration poll interval setting (elastic#26739) [Filebeat] Replace copy_from with templated value (elastic#26631) Reduce the verbosity of the debug log for monitoring (elastic#26583) Add instructions on testing metricbeat kubernetes module (elastic#26643) Revert "[CI] fight the flakiness with some retry option in the CI only for the Pull Requests (elastic#26617)" (elastic#26704) Packaging: linux/armv7 is not supported (elastic#26706) Cyberarkpas: Link to official docs on how to setup TLS (elastic#26614) ...

andrewkroh · 2021-09-09T13:57:36Z

@Mergifyio backport 7.x

…s of Elasticsearch (#26737) Improve the pipeline/compatibility code so that all processors in a pipeline are scanned and acted-upon to ensure compatibility. This means: - Scan processors in on_failure section (both the pipeline's and each individual processor on-failure handler). - Scan the inner processor in a foreach processor. Add a new CI stage, module-compat-7.11, to filebeat/Jenkinsfile.yml and x-pack/filebeat/Jenkinsfile.yml, in order to run Filebeat modules tests under ES 7.11.2 to ensure that all pipelines are functional. This test uses a new flag, TESTING_FILEBEAT_SKIP_DIFF, to instruct the integration test to skip the comparison between the generated documents and the golden/expected files. The test will ensure that the pipeline loads, there are no ingest errors, the fields in the generated documents are valid and the number of returned documents matches the expected. This is intended to avoid having to maintain multiple versions of the golden files due to differences between ES versions and available processors. Also fixes the fortinet and threatintel modules pipelines so that they pass the new test, as some fields were left behind due to the uri_parts processor being removed. (cherry picked from commit 181cf69) # Conflicts: # x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml # x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml

mergify · 2021-09-09T13:58:29Z

Command backport 7.x: success

Backports have been created

#27841 Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch (backport #26737) has been created for branch 7.x

…s of Elasticsearch (backport #26737) (#27841) * Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch (#26737) Improve the pipeline/compatibility code so that all processors in a pipeline are scanned and acted-upon to ensure compatibility. This means: - Scan processors in on_failure section (both the pipeline's and each individual processor on-failure handler). - Scan the inner processor in a foreach processor. Add a new CI stage, module-compat-7.11, to filebeat/Jenkinsfile.yml and x-pack/filebeat/Jenkinsfile.yml, in order to run Filebeat modules tests under ES 7.11.2 to ensure that all pipelines are functional. This test uses a new flag, TESTING_FILEBEAT_SKIP_DIFF, to instruct the integration test to skip the comparison between the generated documents and the golden/expected files. The test will ensure that the pipeline loads, there are no ingest errors, the fields in the generated documents are valid and the number of returned documents matches the expected. This is intended to avoid having to maintain multiple versions of the golden files due to differences between ES versions and available processors. Also fixes the fortinet and threatintel modules pipelines so that they pass the new test, as some fields were left behind due to the uri_parts processor being removed. (cherry picked from commit 181cf69) * Remove script.cache.max_size / script.max_compilations_rate from Filebeat tests (#19896) These setting has been removed as per elastic/elasticsearch#59262 causing the ES container to not launch. elasticsearch_1 | java.lang.IllegalArgumentException: unknown setting [script.cache.max_size] please check that any required plugins are installed, or check the breaking changes documentation for removed settings * Configure ingest node options in env and not in FB test_modules.py Co-authored-by: Adrian Serrano <[email protected]> Co-authored-by: Andrew Kroh <[email protected]>

adriansr added 2 commits July 6, 2021 15:28

ES Compat: Evaluate processors in all pipeline flows

dee5cbf

Test Filebeat modules for 7.11 compatibility

3dc85df

adriansr requested a review from v1v July 6, 2021 13:42

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 6, 2021

adriansr marked this pull request as draft July 6, 2021 13:42

adriansr added Filebeat Filebeat in progress Pull request is currently in progress. Team:Security-External Integrations labels Jul 6, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 6, 2021

Correct test filter

5447dab

v1v approved these changes Jul 6, 2021

View reviewed changes

adriansr added 4 commits July 6, 2021 17:33

Try to figure out why tests are passing when they shouldn't

9357409

Fix yaml multiline string in stage definition

41edfcf

Revert "Try to figure out why tests are passing when they shouldn't"

b419add

This reverts commit 9357409.

adriansr marked this pull request as ready for review July 7, 2021 09:06

adriansr added review and removed in progress Pull request is currently in progress. labels Jul 7, 2021

andrewkroh approved these changes Jul 7, 2021

View reviewed changes

andrewkroh mentioned this pull request Jul 7, 2021

[Filebeat] Module incompatibility with older ES/Kibana versions #26629

Closed

Change logger key name to snake case

9ee4d68

adriansr merged commit 181cf69 into elastic:master Jul 7, 2021

adriansr deleted the fb_pipeline_compat branch July 7, 2021 13:30

mergify bot mentioned this pull request Sep 9, 2021

Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch (backport #26737) #27841

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch #26737

Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch #26737

adriansr commented Jul 6, 2021

elasticmachine commented Jul 6, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Jul 7, 2021

andrewkroh left a comment

andrewkroh Jul 7, 2021

elasticmachine commented Jul 7, 2021

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

andrewkroh commented Sep 9, 2021

mergify bot commented Sep 9, 2021

Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch #26737

Filebeat: Ensure module pipelines compatibility with previous versions of Elasticsearch #26737

Conversation

adriansr commented Jul 6, 2021

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Jul 6, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Jul 7, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh Jul 7, 2021

Choose a reason for hiding this comment

elasticmachine commented Jul 7, 2021

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

andrewkroh commented Sep 9, 2021

mergify bot commented Sep 9, 2021

elasticmachine commented Jul 6, 2021 •

edited by jenkins-beats-ci bot

Loading