Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restrict synthetics/browser from fleet #25831

Closed
wants to merge 5 commits into from

Conversation

andrewvc
Copy link
Contributor

@andrewvc andrewvc commented May 24, 2021

We do not yet have a synthetics fleet image available, so, for now, users should not be allowed to create synthetics inputs in fleet.

@michalpristas can you verify that this is the right place for this file to exist? Once we have a special big docker image with synthetics deps we can conditionally remove this restriction in that special image (which has access to real browsers and graphics libs).

Fixes #25775
See also #22932 (comment)

Checklist

- [ ] My code follows the style guidelines of this project

  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Not yet possible until elastic/uptime#276 is ready

We do not yet have a synthetics fleet image available, so, for now,
users should not be allowed to create synthetics inputs in fleet.

See elastic#22932 (comment)
@andrewvc andrewvc added enhancement Team:obs-ds-hosted-services Label for the Observability Hosted Services team Agent labels May 24, 2021
@andrewvc andrewvc self-assigned this May 24, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 24, 2021
@andrewvc andrewvc requested a review from michalpristas May 24, 2021 19:13
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 24, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #25831 updated

  • Start Time: 2021-06-08T21:27:48.568+0000

  • Duration: 131 min 33 sec

  • Commit: 0092644

Test stats 🧪

Test Results
Failed 0
Passed 47421
Skipped 5253
Total 52674

Trends 🧪

Image of Build Times

Image of Tests

Log output

Expand to view the last 100 lines of log output

[2021-06-08T23:37:02.562Z]  Go version:        go1.13.15
[2021-06-08T23:37:02.562Z]  Git commit:        48d30b5
[2021-06-08T23:37:02.562Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-06-08T23:37:02.562Z]  OS/Arch:           linux/amd64
[2021-06-08T23:37:02.562Z]  Context:           default
[2021-06-08T23:37:02.562Z]  Experimental:      true
[2021-06-08T23:37:02.562Z] 
[2021-06-08T23:37:02.562Z] Server: Docker Engine - Community
[2021-06-08T23:37:02.562Z]  Engine:
[2021-06-08T23:37:02.562Z]   Version:          20.10.3
[2021-06-08T23:37:02.562Z]   API version:      1.41 (minimum version 1.12)
[2021-06-08T23:37:02.562Z]   Go version:       go1.13.15
[2021-06-08T23:37:02.562Z]   Git commit:       46229ca
[2021-06-08T23:37:02.562Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-06-08T23:37:02.562Z]   OS/Arch:          linux/amd64
[2021-06-08T23:37:02.562Z]   Experimental:     false
[2021-06-08T23:37:02.562Z]  containerd:
[2021-06-08T23:37:02.562Z]   Version:          1.4.3
[2021-06-08T23:37:02.562Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-06-08T23:37:02.562Z]  runc:
[2021-06-08T23:37:02.562Z]   Version:          1.0.0-rc92
[2021-06-08T23:37:02.562Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-06-08T23:37:02.562Z]  docker-init:
[2021-06-08T23:37:02.562Z]   Version:          0.19.0
[2021-06-08T23:37:02.562Z]   GitCommit:        de40ad0
[2021-06-08T23:37:02.562Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-06-08T23:37:02.562Z] Unable to find image 'alpine:3.4' locally
[2021-06-08T23:37:03.500Z] 3.4: Pulling from library/alpine
[2021-06-08T23:37:03.761Z] c1e54eec4b57: Pulling fs layer
[2021-06-08T23:37:04.023Z] c1e54eec4b57: Verifying Checksum
[2021-06-08T23:37:04.023Z] c1e54eec4b57: Download complete
[2021-06-08T23:37:04.283Z] c1e54eec4b57: Pull complete
[2021-06-08T23:37:04.284Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-06-08T23:37:04.284Z] Status: Downloaded newer image for alpine:3.4
[2021-06-08T23:37:06.198Z] Change permissions with write access of all files inside the specific folder
[2021-06-08T23:37:06.792Z] Running in /var/lib/jenkins/workspace/PR-25831-4-d1d76fa0-41f7-47c8-a397-77af4f1263eb/src/github.com/elastic/beats/build
[2021-06-08T23:37:07.094Z] + rm -rf ve
[2021-06-08T23:37:07.094Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-06-08T23:37:07.417Z] + python .ci/scripts/pre_archive_test.py
[2021-06-08T23:37:17.430Z] Copy ./x-pack/elastic-agent/build into build/x-pack/elastic-agent/build
[2021-06-08T23:37:17.430Z] Copy ./x-pack/elastic-agent/build/golang-crossbuild into build/x-pack/elastic-agent/build/golang-crossbuild
[2021-06-08T23:37:17.430Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build
[2021-06-08T23:37:17.430Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build
[2021-06-08T23:37:17.430Z] Copy ./x-pack/metricbeat/build into build/x-pack/metricbeat/build
[2021-06-08T23:37:17.430Z] Copy ./x-pack/metricbeat/build/golang-crossbuild into build/x-pack/metricbeat/build/golang-crossbuild
[2021-06-08T23:37:17.430Z] Copy ./x-pack/heartbeat/build into build/x-pack/heartbeat/build
[2021-06-08T23:37:17.430Z] Copy ./x-pack/heartbeat/build/golang-crossbuild into build/x-pack/heartbeat/build/golang-crossbuild
[2021-06-08T23:37:17.430Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-06-08T23:37:17.430Z] Copy ./x-pack/filebeat/build/golang-crossbuild into build/x-pack/filebeat/build/golang-crossbuild
[2021-06-08T23:37:17.449Z] Running in /var/lib/jenkins/workspace/PR-25831-4-d1d76fa0-41f7-47c8-a397-77af4f1263eb/src/github.com/elastic/beats/build
[2021-06-08T23:37:17.467Z] Recording test results
[2021-06-08T23:37:20.158Z] None of the test reports contained any result
[2021-06-08T23:37:20.167Z] [Checks API] No suitable checks publisher found.
[2021-06-08T23:37:20.522Z] + go clean -modcache
[2021-06-08T23:37:24.146Z] Cleaning up /var/lib/jenkins/workspace/PR-25831-4-d1d76fa0-41f7-47c8-a397-77af4f1263eb
[2021-06-08T23:37:24.146Z] Client: Docker Engine - Community
[2021-06-08T23:37:24.146Z]  Version:           20.10.3
[2021-06-08T23:37:24.146Z]  API version:       1.41
[2021-06-08T23:37:24.146Z]  Go version:        go1.13.15
[2021-06-08T23:37:24.146Z]  Git commit:        48d30b5
[2021-06-08T23:37:24.146Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-06-08T23:37:24.146Z]  OS/Arch:           linux/amd64
[2021-06-08T23:37:24.146Z]  Context:           default
[2021-06-08T23:37:24.146Z]  Experimental:      true
[2021-06-08T23:37:24.146Z] 
[2021-06-08T23:37:24.146Z] Server: Docker Engine - Community
[2021-06-08T23:37:24.146Z]  Engine:
[2021-06-08T23:37:24.147Z]   Version:          20.10.3
[2021-06-08T23:37:24.147Z]   API version:      1.41 (minimum version 1.12)
[2021-06-08T23:37:24.147Z]   Go version:       go1.13.15
[2021-06-08T23:37:24.147Z]   Git commit:       46229ca
[2021-06-08T23:37:24.147Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-06-08T23:37:24.147Z]   OS/Arch:          linux/amd64
[2021-06-08T23:37:24.147Z]   Experimental:     false
[2021-06-08T23:37:24.147Z]  containerd:
[2021-06-08T23:37:24.147Z]   Version:          1.4.3
[2021-06-08T23:37:24.147Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-06-08T23:37:24.147Z]  runc:
[2021-06-08T23:37:24.147Z]   Version:          1.0.0-rc92
[2021-06-08T23:37:24.147Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-06-08T23:37:24.147Z]  docker-init:
[2021-06-08T23:37:24.147Z]   Version:          0.19.0
[2021-06-08T23:37:24.147Z]   GitCommit:        de40ad0
[2021-06-08T23:37:24.147Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-06-08T23:38:10.868Z] Change permissions with write access of all files inside the specific folder
[2021-06-08T23:38:10.898Z] Running in /var/lib/jenkins/workspace/PR-25831-4-d1d76fa0-41f7-47c8-a397-77af4f1263eb
[2021-06-08T23:38:15.517Z] + gsutil --version
[2021-06-08T23:38:17.472Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-08T23:38:17.782Z] + gcloud auth activate-service-account --key-file ****
[2021-06-08T23:38:18.351Z] Activated service account credentials for: [[email protected]]
[2021-06-08T23:38:18.927Z] + gsutil -m -q cp -a public-read eC1wYWNrL2VsYXN0aWMtYWdlbnQtcGFja2FnaW5nLWxpbnV4MDA5MjY0NDIwYjllZDQ3YWU0MWI3Njg1YmYwZmQxOTQ4MTEzOTUyNA gs://beats-ci-temp/ci/cache/
[2021-06-08T23:38:20.966Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-08T23:38:21.034Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25831/src/github.com/elastic/beats
[2021-06-08T23:38:21.345Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25831
[2021-06-08T23:38:21.404Z] [INFO] getVaultSecret: Getting secrets
[2021-06-08T23:38:21.452Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-08T23:38:22.159Z] + chmod 755 generate-build-data.sh
[2021-06-08T23:38:22.159Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25831/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25831/runs/4 FAILURE 7833325
[2021-06-08T23:38:22.159Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25831/runs/4/steps/?limit=10000 -o steps-info.json
[2021-06-08T23:38:34.255Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25831/runs/4/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 47421
Skipped 5253
Total 52674

@michalpristas
Copy link
Contributor

location is fine but if you want to pack it into a resulting package packages.yml needs to be updated

@andrewvc
Copy link
Contributor Author

andrewvc commented Jun 8, 2021

@michalpristas updated per your suggestions. Is there any way we can test this PR prior to rolling out the fleet package?

@@ -45,6 +45,10 @@ shared:
source: 'elastic-agent.yml'
mode: 0600
config: true
/etc/{{.BeatName}}/capabilities.yml:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to be added in multiple places.
try searching for 'elastic-agent.yml' there should be 5 occurences, you have 1 so far.

@mergify
Copy link
Contributor

mergify bot commented Jun 18, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b restrict-synthetics upstream/restrict-synthetics
git merge upstream/master
git push upstream restrict-synthetics

@ruflin
Copy link
Member

ruflin commented Aug 13, 2021

I wonder if this is the right approach to follow. So far we don't have any capabilities file shipped by default and it is up to the user to create one if needed. No matter what the user configures in the capabilities.yml, synthetics/browser will not work so it should be hardcoded in the build. My understanding is the the only build that supports synthetics/browser is the complete Dockerfile. Instead of doing modifications to the capabilities, instead heartbeat should return an error if this input is sent down and explain to the user that it is not supported. It is not up to Elastic Agent to know if the input is working in a certain environment or not but heartbeat.

My proposal is to move this implementation to heartbeat and not touch capabilities.yml file as it is not designed for this use case.

@andrewvc
Copy link
Contributor Author

Closing this based on Ruflin's concerns and the fact that we already do surface an error message in heartbeat interviews to run browser-based monitors if we are not running on an image with the synthetics dependencies.

@andrewvc andrewvc closed this Aug 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Agent enhancement Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Heartbeat] Synthetics should be on blocklist for plain heartbeat
4 participants