Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate whether a data stream timestamp has been specified in a document #58119

Closed
wants to merge 26 commits into from
Closed

Validate whether a data stream timestamp has been specified in a document #58119

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
de62b05
Validate whether a data stream timestamp has been specified in a docu…
martijnvg Jun 15, 2020
1d7d5ac
no need to lookup data streams in upgrade service, no documents are b…
martijnvg Jun 15, 2020
b76ac27
use the right date field from the huge twitter docs setup
martijnvg Jun 15, 2020
21264e0
removed unused import
martijnvg Jun 15, 2020
a158a8a
Removed other unneeded changes
martijnvg Jun 15, 2020
99f4ba8
fixed more rest tests
martijnvg Jun 15, 2020
d6b1efa
fixed ilm integ test
martijnvg Jun 15, 2020
fd56292
fixed another test
martijnvg Jun 15, 2020
9ed3369
overload constructors to reduce changes in other files.
martijnvg Jun 15, 2020
3d1920a
iter
martijnvg Jun 15, 2020
5cb17c4
undo unrelated changes
martijnvg Jun 15, 2020
2457562
undo another unrelated change
martijnvg Jun 15, 2020
1def3e4
Merge remote-tracking branch 'es/master' into validate_documents_have…
martijnvg Jun 17, 2020
22e80a1
re-order constructor arguments
martijnvg Jun 17, 2020
91b6bbd
added rest yaml test
martijnvg Jun 17, 2020
3064a6b
Merge remote-tracking branch 'es/master' into validate_documents_have…
martijnvg Jun 18, 2020
5883e83
iter
martijnvg Jun 18, 2020
6ae8bc4
adjusted assertion
martijnvg Jun 18, 2020
3759560
adjusted unneeded line changes
martijnvg Jun 18, 2020
f272148
don't validate timestamp field for tombstone documents,
martijnvg Jun 18, 2020
16ce263
Merge remote-tracking branch 'es/master' into validate_documents_have…
martijnvg Jun 18, 2020
0c3cdcb
Merge remote-tracking branch 'es/master' into validate_documents_have…
martijnvg Jun 24, 2020
b01d89a
iter
martijnvg Jun 24, 2020
57bd047
fixed test
martijnvg Jun 24, 2020
3bc6e04
added unit tests
martijnvg Jun 24, 2020
6627f59
fixed tests
martijnvg Jun 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/reference/indices/rollover-index.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,14 +233,14 @@ PUT _index_template/template
"template": {
"mappings": {
"properties": {
"@timestamp": {
"date": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find the original name better, since it complies with ECS and also date opens up for a bit of confusion between field name and type.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, but in the test data sets that is generated (huge twitter setup), has its timestamp in the date field. I think this should be changed in a followup change?

"type": "date"
}
}
}
},
"data_stream": {
"timestamp_field": "@timestamp"
"timestamp_field": "date"
}
}
-----------------------------------
Expand Down
12 changes: 9 additions & 3 deletions modules/reindex/src/test/resources/rest-api-spec/test/reindex/96_data_streams.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ teardown:
index:
index: logs-foobar
refresh: true
body: { foo: bar }
body:
foo: bar
timestamp: '2020-12-12'

- do:
reindex:
Expand Down Expand Up @@ -65,7 +67,9 @@ teardown:
index:
index: old-logs-index
refresh: true
body: { foo: bar }
body:
foo: bar
timestamp: '2020-12-12'

- do:
reindex:
Expand Down Expand Up @@ -96,7 +100,9 @@ teardown:
index:
index: logs-foobar
refresh: true
body: { foo: bar }
body:
foo: bar
timestamp: '2020-12-12'

- do:
reindex:
Expand Down
2 changes: 1 addition & 1 deletion ...murmur3/src/test/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapperTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ public void setup() {
return indexService.newQueryShardContext(0, null, () -> { throw new UnsupportedOperationException(); }, null);
};
parser = new DocumentMapperParser(indexService.getIndexSettings(), indexService.mapperService(), indexService.xContentRegistry(),
indexService.similarityService(), mapperRegistry, queryShardContext);
indexService.similarityService(), mapperRegistry, queryShardContext, null);
}

@Override
Expand Down
69 changes: 63 additions & 6 deletions rest-api-spec/src/main/resources/rest-api-spec/test/indices.data_stream/10_basic.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,9 @@ setup:
- do:
index:
index: simple-data-stream1
body: { foo: bar }
body:
'@timestamp': '2020-12-12'
foo: bar

- do:
indices.refresh:
Expand Down Expand Up @@ -241,27 +243,27 @@ setup:
- do:
index:
index: logs-foobar
body: { foo: bar }
body: { timestamp: '2020-12-12' }
- match: { _index: .ds-logs-foobar-000001 }

- do:
catch: bad_request
index:
index: .ds-logs-foobar-000001
body: { foo: bar }
body: { timestamp: '2020-12-12' }

- do:
bulk:
body:
- create:
_index: .ds-logs-foobar-000001
- foo: bar
- timestamp: '2020-12-12'
- index:
_index: .ds-logs-foobar-000001
- foo: bar
- timestamp: '2020-12-12'
- create:
_index: logs-foobar
- foo: bar
- timestamp: '2020-12-12'
- match: { errors: true }
- match: { items.0.create.status: 400 }
- match: { items.0.create.error.type: illegal_argument_exception }
Expand All @@ -276,3 +278,58 @@ setup:
indices.delete_data_stream:
name: logs-foobar
- is_true: acknowledged

---
"Indexing a document into a data stream without a timestamp field":
- skip:
version: " - 7.9.99"
reason: "enable in 7.9+ when backported"
features: allowed_warnings

- do:
allowed_warnings:
- "index template [generic_logs_template] has index patterns [logs-*] matching patterns from existing older templates [global] with patterns (global => [*]); this template [generic_logs_template] will take precedence during new index creation"
indices.put_index_template:
name: generic_logs_template
body:
index_patterns: logs-*
template:
mappings:
properties:
'timestamp':
type: date
data_stream:
timestamp_field: timestamp

- do:
catch: bad_request
index:
index: logs-foobar
body: { foo: bar }

- do:
bulk:
body:
- create:
_index: logs-foobar
- foo: bar
- create:
_index: logs-foobar
- timestamp: '2020-12-12'
- create:
_index: logs-foobar
- timestamp: ['2020-12-12', '2022-12-12']
- match: { errors: true }
- match: { items.0.create.status: 400 }
- match: { items.0.create.error.type: illegal_argument_exception }
- match: { items.0.create.error.reason: "data stream timestamp field [timestamp] is missing" }
- match: { items.1.create.result: created }
- match: { items.1.create._index: .ds-logs-foobar-000001 }
- match: { items.2.create.status: 400 }
- match: { items.2.create.error.caused_by.type: illegal_argument_exception }
- match: { items.2.create.error.caused_by.reason: "data stream timestamp field [timestamp] encountered multiple values" }

- do:
indices.delete_data_stream:
name: logs-foobar
- is_true: acknowledged
4 changes: 3 additions & 1 deletion ...pi-spec/src/main/resources/rest-api-spec/test/indices.data_stream/20_unsupported_apis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@
index:
index: logs-foobar
refresh: true
body: { foo: bar }
body:
'@timestamp': '2020-12-12'
foo: bar
- match: {_index: .ds-logs-foobar-000001}

- do:
Expand Down
4 changes: 3 additions & 1 deletion .../src/main/resources/rest-api-spec/test/indices.data_stream/30_auto_create_data_stream.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@
index:
index: logs-foobar
refresh: true
body: { foo: bar }
body:
'timestamp': '2020-12-12'
foo: bar

- do:
search:
Expand Down
28 changes: 14 additions & 14 deletions server/src/internalClusterTest/java/org/elasticsearch/action/bulk/BulkIntegrationIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,28 +233,28 @@ public void testMixedAutoCreate() throws Exception {
client().execute(PutComposableIndexTemplateAction.INSTANCE, createTemplateRequest).actionGet();

BulkRequest bulkRequest = new BulkRequest();
bulkRequest.add(new IndexRequest("logs-foobar").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobaz").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barbaz").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barfoo").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobar").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobaz").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barbaz").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barfoo").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
BulkResponse bulkResponse = client().bulk(bulkRequest).actionGet();
assertThat("bulk failures: " + Strings.toString(bulkResponse), bulkResponse.hasFailures(), is(false));

bulkRequest = new BulkRequest();
bulkRequest.add(new IndexRequest("logs-foobar").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobaz2").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barbaz").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barfoo2").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobar").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobaz2").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barbaz").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barfoo2").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkResponse = client().bulk(bulkRequest).actionGet();
assertThat("bulk failures: " + Strings.toString(bulkResponse), bulkResponse.hasFailures(), is(false));

bulkRequest = new BulkRequest();
bulkRequest.add(new IndexRequest("logs-foobar").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobaz2").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobaz3").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barbaz").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barfoo2").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barfoo3").opType(CREATE).source("{}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobar").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobaz2").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-foobaz3").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barbaz").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barfoo2").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkRequest.add(new IndexRequest("logs-barfoo3").opType(CREATE).source("{\"@timestamp\": \"2020-12-12\"}", XContentType.JSON));
bulkResponse = client().bulk(bulkRequest).actionGet();
assertThat("bulk failures: " + Strings.toString(bulkResponse), bulkResponse.hasFailures(), is(false));

Expand Down
Loading