Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Synthetics] Document changing retention on a per-data-stream basis #1944

Merged
merged 4 commits into from
Aug 22, 2022
Merged

[Synthetics] Document changing retention on a per-data-stream basis #1944

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
03c479c
[Synthetics] Document changing retention on a per-data-stream basis
andrewvc Jun 23, 2022
8366520
PR feedback
andrewvc Jun 23, 2022
0d925cc
restructure to give context then hand off to fleet doc
colleenmcginnis Aug 16, 2022
e4fce65
clean up
colleenmcginnis Aug 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/en/observability/index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@ include::inspect-uptime-duration-anomalies.asciidoc[leveloffset=+3]

include::configure-uptime-settings.asciidoc[leveloffset=+2]

include::synthetics-manage-retention.asciidoc[leveloffset=+2]

include::troubleshoot-uptime-mapping-issues.asciidoc[leveloffset=+2]

// User experience
Expand Down
50 changes: 50 additions & 0 deletions docs/en/observability/synthetics-manage-retention.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
[[synthetics-manage-retention]]
= Manage data retention

When you set up a synthetic monitor, data from the monitor is saved in
https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html[{es} data streams],
an append-only structure in {es}.

There are six data streams recorded by synthetic monitors: `http`, `tcp`, `icmp`, `browser`, `browser.network`, `browser.screenshot`.
Elastic will retain data from each data stream for some time period,
and the default time period varies by data stream.
If you want to reduce the amount of storage required or store data for longer,
you can customize how long to retain data for each data stream.

[discrete]
[[synthetics-manage-retention-data-streams]]
== Synthetics data streams

There are six data streams recorded by synthetic monitors:

[options="header"]
|===
| Data stream | Data includes | Default retention period |
| `http` | The URL that was checked, the status of the check, and any errors that occurred | 1 year |
| `tcp` | The URL that was checked, the status of the check, and any errors that occurred | 1 year |
| `icmp` | The URL that was checked, the status of the check, and any errors that occurred | 1 year |
| `browser` | The URL that was checked, the status of the check, and any errors that occurred | 1 year |
| `browser.screenshot` | Binary image data used to construct a screenshot and metadata with information related to de-duplicating this data | 14 days |
| `browser.network` | Detailed metadata around requests for resources required by the pages being checked | 14 days |
|===

All types of checks record core metadata.
Browser-based checks store two additional types of data: network and screenshot documents.
These browser-specific indices are usually many times larger than the core metadata.
The relative sizes of each vary depending on the sites being
checked with network data usually being the larger of the two by a significant factor.

[discrete]
[[synthetics-manage-retention-customize]]
== Customize data stream lifecycles

If Synthetics browser data streams are storing data longer than necessary,
you can opt to retain data for a shorter period.

To find Synthetics data streams:

. Navigate to https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html[{kib} index management].
. Filter the list of data streams for those containing the term `synthetics`.
.. In the UI there will be three types of browser data streams: `synthetics-browser-*`, `synthetics-browser.network-*`, and `synthetics-browser.screenshot-*`.

Then, you can refer to https://www.elastic.co/guide/en/fleet/current/data-streams.html#data-streams-ilm-tutorial[Tutorial: Customize data retention for integrations] to learn how to apply a custom {ilm-init} policy to the browser data streams.