Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "synthetics-*-*" templates for synthetics fleet data #62193

Merged
merged 6 commits into from
Sep 14, 2020
Merged

Add "synthetics-*-*" templates for synthetics fleet data #62193

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
af938d8
Add "synthetics-*-*" templates for synthetics fleet data
dakrone Sep 9, 2020
ba48b67
Include synthetics things in isXPackTemplate method
dakrone Sep 10, 2020
9fab29f
Merge remote-tracking branch 'origin/master' into add-synthetics-temp…
dakrone Sep 14, 2020
d21aaac
Unset index.query.default_field
dakrone Sep 14, 2020
c17267f
Remove unneeded 'host' field
dakrone Sep 14, 2020
5f9fb0f
Merge branch 'master' into add-synthetics-templates
elasticmachine Sep 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/reference/data-streams/set-up-a-data-stream.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ template for a data stream must specify:

[IMPORTANT]
====
{es} has built-in index templates for the `metrics-*-*` and `logs-*-*` index
{es} has built-in index templates for the `metrics-*-*`, `logs-*-*`, and `synthetics-*-*` index
patterns, each with a priority of `100`.
{ingest-guide}/ingest-management-overview.html[{agent}] uses these templates to
create data streams. If you use {agent}, assign your index templates a priority
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/docs/index_.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ the operation automatically creates the index and applies any matching

[IMPORTANT]
====
{es} has built-in index templates for the `metrics-*-*` and `logs-*-*` index
{es} has built-in index templates for the `metrics-*-*`, `logs-*-*`, and `synthetics-*-*` index
patterns, each with a priority of `100`.
{ingest-guide}/ingest-management-overview.html[{agent}] uses these templates to
create data streams. If you use {agent}, assign your index templates a priority
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/indices/index-templates.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ If a new data stream or index matches more than one index template, the index te

[IMPORTANT]
====
{es} has built-in index templates for the `metrics-*-*` and `logs-*-*` index
{es} has built-in index templates for the `metrics-*-*`, `logs-*-*`, and `synthetics-*-*` index
patterns, each with a priority of `100`.
{ingest-guide}/ingest-management-overview.html[{agent}] uses these templates to
create data streams. If you use {agent}, assign your index templates a priority
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/indices/put-index-template.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ used to match the names of data streams and indices during creation.
+
[IMPORTANT]
====
{es} has built-in index templates for the `metrics-*-*` and `logs-*-*` index
{es} has built-in index templates for the `metrics-*-*`, `logs-*-*`, and `synthetics-*-*` index
patterns, each with a priority of `100`.
{ingest-guide}/ingest-management-overview.html[{agent}] uses these templates to
create data streams. If you use {agent}, assign your index templates a priority
Expand Down
3 changes: 3 additions & 0 deletions test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1276,6 +1276,9 @@ protected static boolean isXPackTemplate(String name) {
case "metrics":
case "metrics-settings":
case "metrics-mappings":
case "synthetics":
case "synthetics-settings":
case "synthetics-mappings":
case ".snapshot-blob-cache":
return true;
default:
Expand Down
57 changes: 57 additions & 0 deletions x-pack/plugin/core/src/main/resources/synthetics-mappings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"template": {
"mappings": {
"dynamic_templates": [
{
"strings_as_keyword": {
"mapping": {
"ignore_above": 1024,
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"date_detection": false,
"properties": {
"@timestamp": {
"type": "date"
},
"data_stream": {
"properties": {
"type": {
"type": "constant_keyword",
"value": "synthetics"
},
"dataset": {
"type": "constant_keyword"
},
"namespace": {
"type": "constant_keyword"
}
}
},
"ecs": {
"properties": {
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"host": {
dakrone marked this conversation as resolved.
Show resolved Hide resolved
"properties": {
"ip": {
"type": "ip"
}
}
}
}
}
},
"_meta": {
"description": "default mappings for the synthetics index template installed by x-pack",
"managed": true
},
"version": ${xpack.stack.template.version}
}
12 changes: 12 additions & 0 deletions x-pack/plugin/core/src/main/resources/synthetics-policy.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "50gb",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewvc I assume this also works for synthetics for now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH we haven't done enough benchmarking to know, but it seems reasonable

"max_age": "30d"
}
}
}
}
}
20 changes: 20 additions & 0 deletions x-pack/plugin/core/src/main/resources/synthetics-settings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "synthetics"
},
"codec": "best_compression",
"query": {
"default_field": ["message"]
dakrone marked this conversation as resolved.
Show resolved Hide resolved
}
}
}
},
"_meta": {
"description": "default settings for the synthetics index template installed by x-pack",
"managed": true
},
"version": ${xpack.stack.template.version}
}
14 changes: 14 additions & 0 deletions x-pack/plugin/core/src/main/resources/synthetics-template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"index_patterns": ["synthetics-*-*"],
"priority": 100,
"data_stream": {},
"composed_of": [
"synthetics-mappings",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpountz I remember you wanted to "not" use components here. Perhaps it is a good time to adjust?

"synthetics-settings"
],
"_meta": {
"description": "default synthetics template installed by x-pack",
"managed": true
},
"version": ${xpack.stack.template.version}
}
39 changes: 36 additions & 3 deletions x-pack/plugin/stack/src/main/java/org/elasticsearch/xpack/stack/StackTemplateRegistry.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,37 @@ public class StackTemplateRegistry extends IndexTemplateRegistry {
TEMPLATE_VERSION_VARIABLE
);

//////////////////////////////////////////////////////////
// Synthetics components (for matching synthetics-*-* indices)
//////////////////////////////////////////////////////////
public static final String SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME = "synthetics-mappings";
public static final String SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME = "synthetics-settings";
public static final String SYNTHETICS_ILM_POLICY_NAME = "synthetics";
public static final String SYNTHETICS_INDEX_TEMPLATE_NAME = "synthetics";

public static final IndexTemplateConfig SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE = new IndexTemplateConfig(
SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
"/synthetics-mappings.json",
REGISTRY_VERSION,
TEMPLATE_VERSION_VARIABLE
);
public static final IndexTemplateConfig SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE = new IndexTemplateConfig(
SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME,
"/synthetics-settings.json",
REGISTRY_VERSION,
TEMPLATE_VERSION_VARIABLE
);
public static final LifecyclePolicyConfig SYNTHETICS_ILM_POLICY = new LifecyclePolicyConfig(
SYNTHETICS_ILM_POLICY_NAME,
"/synthetics-policy.json"
);
public static final IndexTemplateConfig SYNTHETICS_INDEX_TEMPLATE = new IndexTemplateConfig(
SYNTHETICS_INDEX_TEMPLATE_NAME,
"/synthetics-template.json",
REGISTRY_VERSION,
TEMPLATE_VERSION_VARIABLE
);

public StackTemplateRegistry(
Settings nodeSettings,
ClusterService clusterService,
Expand All @@ -104,7 +135,7 @@ public StackTemplateRegistry(
@Override
protected List<LifecyclePolicyConfig> getPolicyConfigs() {
if (stackTemplateEnabled) {
return Arrays.asList(LOGS_ILM_POLICY, METRICS_ILM_POLICY);
return Arrays.asList(LOGS_ILM_POLICY, METRICS_ILM_POLICY, SYNTHETICS_ILM_POLICY);
} else {
return Collections.emptyList();
}
Expand All @@ -117,7 +148,9 @@ protected List<IndexTemplateConfig> getComponentTemplateConfigs() {
LOGS_MAPPINGS_COMPONENT_TEMPLATE,
LOGS_SETTINGS_COMPONENT_TEMPLATE,
METRICS_MAPPINGS_COMPONENT_TEMPLATE,
METRICS_SETTINGS_COMPONENT_TEMPLATE
METRICS_SETTINGS_COMPONENT_TEMPLATE,
SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE,
SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE
);
} else {
return Collections.emptyList();
Expand All @@ -127,7 +160,7 @@ protected List<IndexTemplateConfig> getComponentTemplateConfigs() {
@Override
protected List<IndexTemplateConfig> getComposableTemplateConfigs() {
if (stackTemplateEnabled) {
return Arrays.asList(LOGS_INDEX_TEMPLATE, METRICS_INDEX_TEMPLATE);
return Arrays.asList(LOGS_INDEX_TEMPLATE, METRICS_INDEX_TEMPLATE, SYNTHETICS_INDEX_TEMPLATE);
} else {
return Collections.emptyList();
}
Expand Down
22 changes: 18 additions & 4 deletions .../plugin/stack/src/test/java/org/elasticsearch/xpack/stack/StackTemplateRegistryTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,11 @@ public void testThatNonExistingPoliciesAreAddedImmediately() throws Exception {
final PutLifecycleAction.Request putRequest = (PutLifecycleAction.Request) request;
assertThat(
putRequest.getPolicy().getName(),
anyOf(equalTo(StackTemplateRegistry.LOGS_ILM_POLICY_NAME), equalTo(StackTemplateRegistry.METRICS_ILM_POLICY_NAME))
anyOf(
equalTo(StackTemplateRegistry.LOGS_ILM_POLICY_NAME),
equalTo(StackTemplateRegistry.METRICS_ILM_POLICY_NAME),
equalTo(StackTemplateRegistry.SYNTHETICS_ILM_POLICY_NAME)
)
);
assertNotNull(listener);
return new PutLifecycleAction.Response(true);
Expand All @@ -168,7 +172,7 @@ public void testThatNonExistingPoliciesAreAddedImmediately() throws Exception {

ClusterChangedEvent event = createClusterChangedEvent(Collections.emptyMap(), nodes);
registry.clusterChanged(event);
assertBusy(() -> assertThat(calledTimes.get(), equalTo(2)));
assertBusy(() -> assertThat(calledTimes.get(), equalTo(3)));
}

public void testPolicyAlreadyExists() {
Expand All @@ -180,7 +184,7 @@ public void testPolicyAlreadyExists() {
.stream()
.map(policyConfig -> policyConfig.load(xContentRegistry))
.collect(Collectors.toList());
assertThat(policies, hasSize(2));
assertThat(policies, hasSize(3));
policies.forEach(p -> policyMap.put(p.getName(), p.get()));

client.setVerifier((action, request, listener) -> {
Expand Down Expand Up @@ -209,7 +213,7 @@ public void testPolicyAlreadyExistsButDiffers() throws IOException {
.stream()
.map(policyConfig -> policyConfig.load(xContentRegistry))
.collect(Collectors.toList());
assertThat(policies, hasSize(2));
assertThat(policies, hasSize(3));
policies.forEach(p -> policyMap.put(p.getName(), p.get()));

client.setVerifier((action, request, listener) -> {
Expand Down Expand Up @@ -276,6 +280,8 @@ public void testSameOrHigherVersionTemplateNotUpgraded() {
versions.put(StackTemplateRegistry.LOGS_MAPPINGS_COMPONENT_TEMPLATE_NAME, StackTemplateRegistry.REGISTRY_VERSION);
versions.put(StackTemplateRegistry.METRICS_SETTINGS_COMPONENT_TEMPLATE_NAME, StackTemplateRegistry.REGISTRY_VERSION);
versions.put(StackTemplateRegistry.METRICS_MAPPINGS_COMPONENT_TEMPLATE_NAME, StackTemplateRegistry.REGISTRY_VERSION);
versions.put(StackTemplateRegistry.SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME, StackTemplateRegistry.REGISTRY_VERSION);
versions.put(StackTemplateRegistry.SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME, StackTemplateRegistry.REGISTRY_VERSION);
ClusterChangedEvent sameVersionEvent = createClusterChangedEvent(versions, nodes);
client.setVerifier((action, request, listener) -> {
if (action instanceof PutComponentTemplateAction) {
Expand Down Expand Up @@ -311,6 +317,14 @@ public void testSameOrHigherVersionTemplateNotUpgraded() {
StackTemplateRegistry.METRICS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
StackTemplateRegistry.REGISTRY_VERSION + randomIntBetween(1, 1000)
);
versions.put(
StackTemplateRegistry.SYNTHETICS_SETTINGS_COMPONENT_TEMPLATE_NAME,
StackTemplateRegistry.REGISTRY_VERSION + randomIntBetween(1, 1000)
);
versions.put(
StackTemplateRegistry.SYNTHETICS_MAPPINGS_COMPONENT_TEMPLATE_NAME,
StackTemplateRegistry.REGISTRY_VERSION + randomIntBetween(1, 1000)
);
ClusterChangedEvent higherVersionEvent = createClusterChangedEvent(versions, nodes);
registry.clusterChanged(higherVersionEvent);
}
Expand Down