Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add template for ssl config reference settings #20357

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Jul 30, 2020

What does this PR do?

Use a shared template for the ssl options in reference configs.

I provided an indent function in case this config template needed to be reused
at various indentation levels, but it turns out that all of the uses so far required
an indentation of 2 (so I could have just indented ssl.reference.yml.tmpl manually).

Why is it important?

The reduces duplication of the SSL settings in config templates and ensures that the
reference configs are consistent across outputs and monitoring.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Please review the ssl.reference.yml.tmpl to ensure that it is complete and grammatically correct. It was copied directly from the elasticsearch output reference config.

Related issues

@andrewkroh andrewkroh requested review from a team as code owners July 30, 2020 14:46
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 30, 2020
dev-tools/mage/config.go Outdated Show resolved Hide resolved
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 30, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20357 updated]

  • Start Time: 2020-08-11T12:00:12.288+0000

  • Duration: 128 min 18 sec

Test stats 🧪

Test Results
Failed 1
Passed 15801
Skipped 1922
Total 17724

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat Mac OS X / test_debug_reader – filebeat.tests.system.test_harvester.Test

    • Age: 1
    • Duration: 5.083
    • Error Details: beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 5 seconds.

Steps errors

Expand to view the steps failures

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 9 min 2 sec

    • Start Time: 2020-08-11T12:40:41.933+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 1

    • Duration: 0 min 12 sec

    • Start Time: 2020-08-11T12:50:24.269+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-08-11T14:07:33.917Z]   OS/Arch:          linux/amd64
[2020-08-11T14:07:33.917Z]   Experimental:     false
[2020-08-11T14:07:33.917Z]  containerd:
[2020-08-11T14:07:33.917Z]   Version:          1.2.13
[2020-08-11T14:07:33.917Z]   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
[2020-08-11T14:07:33.917Z]  runc:
[2020-08-11T14:07:33.917Z]   Version:          1.0.0-rc10
[2020-08-11T14:07:33.917Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-08-11T14:07:33.917Z]  docker-init:
[2020-08-11T14:07:33.917Z]   Version:          0.18.0
[2020-08-11T14:07:33.917Z]   GitCommit:        fec3683
[2020-08-11T14:07:34.181Z] Unable to find image 'alpine:3.4' locally
[2020-08-11T14:07:34.760Z] 3.4: Pulling from library/alpine
[2020-08-11T14:07:35.027Z] c1e54eec4b57: Pulling fs layer
[2020-08-11T14:07:35.289Z] c1e54eec4b57: Verifying Checksum
[2020-08-11T14:07:35.289Z] c1e54eec4b57: Download complete
[2020-08-11T14:07:35.557Z] c1e54eec4b57: Pull complete
[2020-08-11T14:07:35.557Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2020-08-11T14:07:35.557Z] Status: Downloaded newer image for alpine:3.4
[2020-08-11T14:07:37.819Z] + python .ci/scripts/pre_archive_test.py
[2020-08-11T14:07:38.080Z] Copy ./libbeat/build into build/libbeat/build
[2020-08-11T14:07:38.093Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/build
[2020-08-11T14:07:38.120Z] Recording test results
[2020-08-11T14:07:40.499Z] Stashed 2 file(s)
[2020-08-11T14:07:40.510Z] Archiving artifacts
[2020-08-11T14:07:41.127Z] + python .ci/scripts/search_system_tests.py
[2020-08-11T14:07:41.237Z] [INFO] system-tests='build/libbeat/build/system-tests'. If no empty then let's create a tarball
[2020-08-11T14:07:41.652Z] + tar --version
[2020-08-11T14:07:41.998Z] + tar --exclude=libbeat--system-tests-linux.tgz -czf libbeat--system-tests-linux.tgz build/libbeat/build/system-tests
[2020-08-11T14:07:44.633Z] Archiving artifacts
[2020-08-11T14:07:48.460Z] Stage "Libbeat crosscompile" skipped due to earlier failure(s)
[2020-08-11T14:07:48.520Z] Stage "Libbeat stress-tests" skipped due to earlier failure(s)
[2020-08-11T14:07:48.635Z] Failed in branch Libbeat
[2020-08-11T14:07:57.881Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-08-11T14:07:57.901Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-08-11T14:07:57.960Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-20357-6/source/source.tgz
[2020-08-11T14:07:57.980Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-20357-6/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-20357/source.tgz
[2020-08-11T14:08:08.435Z] + tar --version
[2020-08-11T14:08:08.734Z] + tar -xpf source.tgz
[2020-08-11T14:08:19.059Z] + rm source.tgz
[2020-08-11T14:08:19.072Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats
[2020-08-11T14:08:19.097Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Lint
[2020-08-11T14:08:19.180Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-08-11T14:08:19.261Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Winlogbeat-oss
[2020-08-11T14:08:19.338Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-08-11T14:08:19.417Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Dockerlogbeat
[2020-08-11T14:08:19.493Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Journalbeat-oss
[2020-08-11T14:08:19.571Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-08-11T14:08:19.648Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-08-11T14:08:19.723Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Packetbeat-Linux
[2020-08-11T14:08:19.798Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-08-11T14:08:19.874Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-08-11T14:08:19.950Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Heartbeat-oss
[2020-08-11T14:08:20.025Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-08-11T14:08:20.099Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-08-11T14:08:20.174Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-08-11T14:08:20.256Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-08-11T14:08:20.347Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-08-11T14:08:20.423Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Libbeat-x-pack
[2020-08-11T14:08:20.498Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-08-11T14:08:20.579Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-08-11T14:08:20.657Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-08-11T14:08:20.749Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-08-11T14:08:20.827Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Filebeat-Windows
[2020-08-11T14:08:20.904Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-08-11T14:08:20.984Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-08-11T14:08:21.060Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-Windows
[2020-08-11T14:08:21.137Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-08-11T14:08:21.218Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-08-11T14:08:21.298Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-08-11T14:08:21.382Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-08-11T14:08:21.458Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-08-11T14:08:21.544Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Filebeat-x-pack
[2020-08-11T14:08:21.618Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-08-11T14:08:21.696Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Filebeat-oss
[2020-08-11T14:08:21.776Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-08-11T14:08:21.856Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-08-11T14:08:21.936Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-08-11T14:08:22.019Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-08-11T14:08:22.096Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Functionbeat-Windows
[2020-08-11T14:08:22.171Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-08-11T14:08:22.263Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Packetbeat-Windows
[2020-08-11T14:08:22.345Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Heartbeat-Windows
[2020-08-11T14:08:22.424Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-08-11T14:08:22.598Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-08-11T14:08:22.681Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-08-11T14:08:22.756Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20357/src/github.com/elastic/beats/Libbeat-oss
[2020-08-11T14:08:23.157Z] + cat
[2020-08-11T14:08:23.157Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-08-11T14:08:23.157Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-08-11T14:08:29.121Z] Body did not finish within grace period; terminating with extreme prejudice
[2020-08-11T14:08:29.122Z] ERROR: runbld post build action failed.
[2020-08-11T14:08:29.123Z] Timeout has been exceeded
[2020-08-11T14:08:29.306Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20357
[2020-08-11T14:08:29.421Z] [INFO] getVaultSecret: Getting secrets
[2020-08-11T14:08:29.508Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-08-11T14:08:30.389Z] + chmod 755 generate-build-data.sh
[2020-08-11T14:08:30.389Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20357/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20357/runs/6 ABORTED 7697841
[2020-08-11T14:08:30.389Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20357/runs/6/steps/?limit=10000 -o steps-info.json
[2020-08-11T14:08:33.617Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20357/runs/6/tests/?status=FAILED -o tests-errors.json

# this allow you to ensure that a specific certificate is used to validate the chain of trust.
#
# The pin is a base64 encoded string of the SHA-256 fingerprint.
#ssl.ca_sha256: ""
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commit looks somewhat bloated due to minor differences between SSL documentation. Maybe it would better to use the config from auditbeat as base for the template?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's bloated because the Elasticsearch output's ssl config (chosen as the source of all ssl config options) was not aligned with logstash, kafka, redis, kibana, and monitoring's config.

@andresrc andresrc added Team:Integrations Label for the Integrations team Team:SIEM labels Jul 31, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 31, 2020
@andrewkroh andrewkroh force-pushed the feature/libbeat/ssl-config-reference-template branch from fbf13b7 to 9b0b9a6 Compare August 3, 2020 22:13
@andrewkroh
Copy link
Member Author

run tests

1 similar comment
@andrewkroh
Copy link
Member Author

run tests

Use a shared template for the ssl options in reference configs. The reduces
duplication of the SSL settings in config templates and ensures that the
reference configs are consistent across outputs and monitoring.

I provided an `indent` function in case this config template needed to be reused
at various indentation levels, but it turns out that all of the uses so far required
an indentation of 2 (so I could have just indented ssl.reference.yml.tmpl manually).
@andrewkroh andrewkroh force-pushed the feature/libbeat/ssl-config-reference-template branch from 9b0b9a6 to 8725ec9 Compare August 11, 2020 11:58
@andrewkroh andrewkroh merged commit 9e4d4f2 into elastic:master Aug 18, 2020
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Aug 24, 2020
* Add template for ssl config reference settings

Use a shared template for the ssl options in reference configs. The reduces
duplication of the SSL settings in config templates and ensures that the
reference configs are consistent across outputs and monitoring.

I provided an `indent` function in case this config template needed to be reused
at various indentation levels, but it turns out that all of the uses so far required
an indentation of 2 (so I could have just indented ssl.reference.yml.tmpl manually).

* Indent any non-whitespace containing lines

(cherry picked from commit 9e4d4f2)
andrewkroh added a commit that referenced this pull request Aug 24, 2020
* Add template for ssl config reference settings

Use a shared template for the ssl options in reference configs. The reduces
duplication of the SSL settings in config templates and ensures that the
reference configs are consistent across outputs and monitoring.

I provided an `indent` function in case this config template needed to be reused
at various indentation levels, but it turns out that all of the uses so far required
an indentation of 2 (so I could have just indented ssl.reference.yml.tmpl manually).

* Indent any non-whitespace containing lines

(cherry picked from commit 9e4d4f2)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
* Add template for ssl config reference settings

Use a shared template for the ssl options in reference configs. The reduces
duplication of the SSL settings in config templates and ensures that the
reference configs are consistent across outputs and monitoring.

I provided an `indent` function in case this config template needed to be reused
at various indentation levels, but it turns out that all of the uses so far required
an indentation of 2 (so I could have just indented ssl.reference.yml.tmpl manually).

* Indent any non-whitespace containing lines
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants