Initial work toward synthetic monitors #21436

andrewvc · 2020-09-30T19:29:56Z

Initial, experimental version of synthetic monitor support for Heartbeat. This works in concert with https://github.com/elastic/synthetics , and is intended to only be run within the docker container built in the synthetics repo. This is guarded by the new ELASTIC_SYNTHETICS_CAPABLE env variable which must be set to true. As a note, this is easy to bypass because the main purpose is to make it clear that we do not support synthetics outside of the docker container for now.

This includes a new directory, sample-synthetics-config in the x-pack directory that demonstrates using heartbeat with synthetics. In the future

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

You can check that this PR works by running (in the x-pack/heartbeat directory):
mage build && env ELASTIC_SYNTHETICS_CAPABLE=true ./heartbeat --path.config sample-synthetics-config/ -e

elasticmachine · 2020-09-30T19:41:42Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #21436 updated
Start Time: 2020-12-18T21:06:52.448+0000
Duration: 102 min 53 sec

Test stats 🧪

Test	Results
Failed	0
Passed	17502
Skipped	1383
Total	18885

Steps errors

Expand to view the steps failures

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 14 sec . View more details on here
Description: terraform apply -auto-approve

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 15 sec . View more details on here
Description: terraform apply -auto-approve

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	17502
Skipped	1383
Total	18885

* ci: add Heartbeat Packaging for Elastic Synthetics job * ci: change to use Ubuntu-18

* ci: add Heartbeat Packaging for Elastic Synthetics job * ci: change to use Ubuntu-18 * ci: use apm GCP credentials

…heartbeat-synthetics

andrewvc · 2020-12-17T00:14:45Z

x-pack/heartbeat/monitors/browser/synthexec/synthexec_test.go

+}
+
+func TestRunCmd(t *testing.T) {
+	cmd := exec.Command("go", "run", "./main.go")


@blakerouse I hope this test addresses your concern re: covering actual binary execution!

Yes it does. Exactly what I was asking for.

blakerouse

This looks great and excited to see it finally landing.

I am interested in getting this to work under Agent without needing a container. Seems playwright works on Linux, Mac, and Windows. I am sure we can come up with a way to bundle it all, not requiring a special container.

blakerouse · 2020-12-17T15:16:00Z

x-pack/heartbeat/monitors/browser/synthexec/synthexec_test.go

+}
+
+func TestRunCmd(t *testing.T) {
+	cmd := exec.Command("go", "run", "./main.go")


Yes it does. Exactly what I was asking for.

blakerouse · 2020-12-17T15:17:40Z

x-pack/heartbeat/seccomp_linux.go

+			"with",
+			"write"}
+
+		fmt.Println("SYSCALLS !!!\n")


I think you forgot to remove this line.

Ah, good catch!

andrewvc · 2020-12-18T14:13:18Z

/packaging

elasticmachine · 2020-12-18T14:13:40Z

Pinging @elastic/uptime (Team:Uptime)

andrewvc · 2020-12-18T14:14:04Z

/packaging test
/test

andrewvc · 2020-12-18T14:15:38Z

/test this

andrewvc · 2020-12-18T14:17:24Z

/packaging this

andrewvc · 2020-12-18T17:06:00Z

/test this

andrewvc · 2020-12-18T17:49:54Z

/test this

v1v · 2020-12-18T18:33:54Z

.ci/heartbeat-synthetics.groovy

@@ -0,0 +1,122 @@
+#!/usr/bin/env groovy


Is this particular pipeline required? The current CI pipeline does not use it, so I wonder if the packaging pipeline should be the one to be used instead?

@kuisathaverat would be the best one to ping here since he added that to this PR. However, I do believe it's essential (today) to make the custom synthetics image based off this branch.

since the x-pack/heartbeat is build in package pipeline, I do not think we need the groovy file. the heartbeat Docker image will be the new synthetics docker image. this pipeline is only to change the name to the Docker image to ...-synthetics

Initial, experimental version of synthetic monitor support for Heartbeat. This works in concert with https://github.com/elastic/synthetics , and is intended to only be run within the docker container built in the synthetics repo. This is guarded by the new ELASTIC_SYNTHETICS_CAPABLE env variable which must be set to true. As a note, this is easy to bypass because the main purpose is to make it clear that we do not support synthetics outside of the docker container for now. This includes a new directory, sample-synthetics-config in the x-pack directory that demonstrates using heartbeat with synthetics. Since we still only want people to run this in the docker image, we guard against accidental use outside of that by checking for the env var ELASTIC_SYNTHETICS_CAPABLE, which we only set in that image (yes, this is easy to circumvent, but we want users to realize they are circumventing something before they do so) (cherry picked from commit e50f673)

…3224) * Initial work toward synthetic monitors (#21436) Initial, experimental version of synthetic monitor support for Heartbeat. This works in concert with https://github.com/elastic/synthetics , and is intended to only be run within the docker container built in the synthetics repo. This is guarded by the new ELASTIC_SYNTHETICS_CAPABLE env variable which must be set to true. As a note, this is easy to bypass because the main purpose is to make it clear that we do not support synthetics outside of the docker container for now. This includes a new directory, sample-synthetics-config in the x-pack directory that demonstrates using heartbeat with synthetics. Since we still only want people to run this in the docker image, we guard against accidental use outside of that by checking for the env var ELASTIC_SYNTHETICS_CAPABLE, which we only set in that image (yes, this is easy to circumvent, but we want users to realize they are circumventing something before they do so) (cherry picked from commit e50f673)

Initial, experimental version of synthetic monitor support for Heartbeat. This works in concert with https://github.com/elastic/synthetics , and is intended to only be run within the docker container built in the synthetics repo. This is guarded by the new ELASTIC_SYNTHETICS_CAPABLE env variable which must be set to true. As a note, this is easy to bypass because the main purpose is to make it clear that we do not support synthetics outside of the docker container for now. This includes a new directory, sample-synthetics-config in the x-pack directory that demonstrates using heartbeat with synthetics. Since we still only want people to run this in the docker image, we guard against accidental use outside of that by checking for the env var ELASTIC_SYNTHETICS_CAPABLE, which we only set in that image (yes, this is easy to circumvent, but we want users to realize they are circumventing something before they do so)

Initial work toward synthetic monitors

2c5b9e2

andrewvc added enhancement Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Sep 30, 2020

andrewvc self-assigned this Sep 30, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 30, 2020

andrewvc mentioned this pull request Sep 30, 2020

ci: add Docker image build and publish elastic/synthetics#63

Merged

andrewvc and others added 22 commits September 30, 2020 18:42

Count errors, report error on monitor.summary

2cfb01a

Handle timestamps that now come as milli floats

c161221

Add mapping for error type

ce5f4b0

update fields.go

1135e1f

Properly add error field

b610e37

ci: add Heartbeat Packaging for Elastic Synthetics job (#21567)

904ba82

ci: change to use Ubuntu-18 (#21569)

0b05d28

* ci: add Heartbeat Packaging for Elastic Synthetics job * ci: change to use Ubuntu-18

ci: use apm GCP credentials (#21571)

0e099f6

* ci: add Heartbeat Packaging for Elastic Synthetics job * ci: change to use Ubuntu-18 * ci: use apm GCP credentials

ci: remove GCP storage (#21575)

d752860

Support new journey/register hook in agent

41643ce

Merge branch 'heartbeat-synthetics' of github.com:elastic/beats into …

7ec3764

…heartbeat-synthetics

Compare float with float for tests

ca989c3

ci: set proper image name

ad421e5

ci: configure downstream job

05bfdc9

ci: remove snapshot from the tag

b756ffa

ci: remove the downstream job

4bcb592

Switch from stdin to inline flag

6cb723b

Merge remote-tracking branch 'origin/master' into heartbeat-synthetics

0c17b22

Update fields.go

172bc3f

Merge remote-tracking branch 'origin/master' into heartbeat-synthetics

e6721ce

Use synthetics directory scanning

4973c6c

Fix fields generation

ca1b3c9

Fix heartbeat.yml perms

b866c95

andrewvc commented Dec 17, 2020

View reviewed changes

blakerouse approved these changes Dec 17, 2020

View reviewed changes

andrewvc added 2 commits December 17, 2020 10:57

Improve var name

434fcbb

Remove unnecessary print

7354f9e

andrewvc marked this pull request as ready for review December 18, 2020 14:13

andrewvc requested review from a team as code owners December 18, 2020 14:13

Merge remote-tracking branch 'origin/master' into heartbeat-synthetics

8c9d592

botelastic bot added the Team:Automation Label for the Observability productivity team label Dec 18, 2020

paulb-elastic added v7.12.0 and removed v7.12.0 labels Dec 18, 2020

v1v reviewed Dec 18, 2020

View reviewed changes

Merge remote-tracking branch 'origin/master' into heartbeat-synthetics

31546e8

andrewvc merged commit e50f673 into master Dec 21, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Dec 21, 2020

andrewvc mentioned this pull request Dec 21, 2020

Cherry-pick #21436 to 7.x: Initial work toward synthetic monitors #23224

Merged

6 tasks

zube bot removed the [zube]: Done label Mar 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial work toward synthetic monitors #21436

Initial work toward synthetic monitors #21436

andrewvc commented Sep 30, 2020 •

edited

Loading

elasticmachine commented Sep 30, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`

Test stats 🧪

andrewvc Dec 17, 2020

blakerouse Dec 17, 2020

blakerouse left a comment

blakerouse Dec 17, 2020

blakerouse Dec 17, 2020

andrewvc Dec 17, 2020

andrewvc commented Dec 18, 2020

elasticmachine commented Dec 18, 2020

andrewvc commented Dec 18, 2020

andrewvc commented Dec 18, 2020

andrewvc commented Dec 18, 2020

andrewvc commented Dec 18, 2020

andrewvc commented Dec 18, 2020

v1v Dec 18, 2020

andrewvc Dec 18, 2020

kuisathaverat Dec 21, 2020

Initial work toward synthetic monitors #21436

Initial work toward synthetic monitors #21436

Conversation

andrewvc commented Sep 30, 2020 • edited Loading

Checklist

How to test this PR locally

elasticmachine commented Sep 30, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Terraform Apply on x-pack/metricbeat/module/aws

Terraform Apply on x-pack/metricbeat/module/aws

💚 Flaky test report

Test stats 🧪

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blakerouse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewvc commented Dec 18, 2020

elasticmachine commented Dec 18, 2020

andrewvc commented Dec 18, 2020

andrewvc commented Dec 18, 2020

andrewvc commented Dec 18, 2020

andrewvc commented Dec 18, 2020

andrewvc commented Dec 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewvc commented Sep 30, 2020 •

edited

Loading

elasticmachine commented Sep 30, 2020 •

edited by jenkins-beats-ci bot

Loading

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`