Skip to content

Commit

Permalink
[Heartbeat] Always use synthetics data streams for browser
Browse files Browse the repository at this point in the history
A frequent UX issue for our users is wanting to set different retention periods for network / screenshot data. Our default settings for browser monitors really hurt here since we lump them all into a `heartbeat` data stream unless users are using fleet. This addresses this issue by defaulting browser monitors to use `synthetics*` data streams which breaks them out by type.

We can't do this in any sort of simple way with the 'heartbeat' prefix.
This shouldn't be a practical issue for most users since in 8.x all
kibana clusters have synthetics templates pre-installed. Users who have
explicitly set their kibana index pattern to only `heartbeat-*` with no
`synthetics-*` will be affected. Given that we are in beta this
potential breakage is worth it for the long term benefits to users.

This also implies Kibana has been setup with Elasticsearch, which is
overwhelmingly the case. If kibana is not setup then the mappings for
this synthetics index will be incorrect, but this is likely not a
real-world concern.

See also elastic/observability-docs#1944
  • Loading branch information
andrewvc committed Jun 23, 2022
1 parent 7b99a8e commit a558406
Show file tree
Hide file tree
Showing 3 changed files with 39 additions and 4 deletions.
16 changes: 15 additions & 1 deletion heartbeat/docs/monitors/monitor-common-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,19 @@ A list of processors to apply to the data generated by the monitor.
See <<filtering-and-enhancing-data>> for information about specifying
processors in your config.

[float]
[[monitor-data-stream]]

Contains options pertaining to data stream naming, following the conventions followed by [Fleet Data Streams](https://www.elastic.co/guide/en/fleet/current/data-streams.html). By default Heartbeat will
write to a datastream named `heartbeat-VERSION` except in the case of `browser` monitors, which will
always follow the Fleet convention of `type-dataset-namespace`.


```yaml
# To enable data streams with the default namespace
data_stream.namespace: default
```

[float]
[[monitor-pipeline]]
===== `pipeline`
Expand All @@ -156,8 +169,9 @@ input is used.

[float]
[[monitor-index]]
===== `index`
===== `index` (deprecated)

This setting is now deprecated in favor of the `data_stream` option.
If present, this formatted string overrides the index for events from this input
(for elasticsearch outputs), or sets the `raw_index` field of the event's
metadata (for other outputs). This string can only refer to the agent name and
Expand Down
9 changes: 9 additions & 0 deletions heartbeat/monitors/factory.go
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,15 @@ func preProcessors(info beat.Info, settings publishSettings, monitorType string)
// Always set event.dataset
procs.AddProcessor(actions.NewAddFields(mapstr.M{"event": mapstr.M{"dataset": dataset}}, true, true))

// always use synthetics data streams for browser monitors, there is no good reason not to
// the default `heartbeat` data stream won't split out network and screenshot data.
// at some point we should make all monitors use the `synthetics` datastreams and retire
// the heartbeat one, but browser is the only beta one, and it would be a breaking change
// to do so otherwise.
if monitorType == "browser" && settings.DataStream == nil {
settings.DataStream = &add_data_stream.DataStream{}
}

if settings.DataStream != nil {
ds := *settings.DataStream
if ds.Type == "" {
Expand Down
18 changes: 15 additions & 3 deletions heartbeat/monitors/factory_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,19 +48,31 @@ func TestPreProcessors(t *testing.T) {
wantProc bool
wantErr bool
}{
"no settings should yield no processor": {
"no settings should yield no processor for lightweight monitor": {
publishSettings{},
"",
nil,
"browser",
"http",
false,
false,
},
"no settings should yield a data stream processor for browsers": {
publishSettings{},
"synthetics-browser-default",
&add_data_stream.DataStream{
Namespace: "default",
Dataset: "browser",
Type: "synthetics",
},
"browser",
true,
false,
},
"exact index should be used exactly": {
publishSettings{Index: *fmtstr.MustCompileEvent("test")},
"test",
nil,
"browser",
"http",
true,
false,
},
Expand Down

0 comments on commit a558406

Please sign in to comment.