Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove support for new indexes using path setting in object/nested fields or index_name in any field #9570

Merged
merged 1 commit into from
Feb 5, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion dev-tools/create-bwc-index.py
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,22 @@ def generate_index(client, version):
'index_analyzer': 'standard',
'search_analyzer': 'keyword',
'search_quote_analyzer': 'english',
}
}
mappings['index_name_and_path'] = {
'properties': {
'parent_multi_field': {
'type': 'string',
'path': 'just_name',
'fields': {
'raw': {'type': 'string', 'index': 'not_analyzed', 'index_name': 'raw_multi_field'}
}
},
'field_with_index_name': {
'type': 'string',
'index_name': 'custom_index_name_for_field'
}
}
}

client.indices.create(index='test', body={
'settings': {
Expand Down
21 changes: 9 additions & 12 deletions docs/reference/indices/get-field-mapping.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,6 @@ You can also use wildcards. The field names can be any of the following:
[horizontal]
Full names:: the full path, including any parent object name the field is
part of (ex. `user.id`).
Index names:: the name of the lucene field (can be different than the
field name if the `index_name` option of the mapping is used).
Field names:: the name of the field without the path to it (ex. `id` for `{ "user" : { "id" : 1 } }`).

The above options are specified in the order the `field` parameter is resolved.
Expand All @@ -76,26 +74,25 @@ For example, consider the following mapping:
"article": {
"properties": {
"id": { "type": "string" },
"title": { "type": "string", "index_name": "text" },
"abstract": { "type": "string", "index_name": "text" },
"title": { "type": "string"},
"abstract": { "type": "string"},
"author": {
"properties": {
"id": { "type": "string" },
"name": { "type": "string", "index_name": "author" }
"name": { "type": "string" }
}
}
}
}
}
--------------------------------------------------

To select the `id` of the `author` field, you can use its full name `author.id`. Using `text` will return
the mapping of `abstract` as it is one of the fields which map to the Lucene field `text`. `name` will return
To select the `id` of the `author` field, you can use its full name `author.id`. `name` will return
the field `author.name`:

[source,js]
--------------------------------------------------
curl -XGET "http://localhost:9200/publications/_mapping/article/field/author.id,text,name"
curl -XGET "http://localhost:9200/publications/_mapping/article/field/author.id,abstract,name"
--------------------------------------------------

returns:
Expand All @@ -105,10 +102,10 @@ returns:
{
"publications": {
"article": {
"text": {
"abstract": {
"full_name": "abstract",
"mapping": {
"abstract": { "type": "string", "index_name": "text" }
"abstract": { "type": "string" }
}
},
"author.id": {
Expand All @@ -120,7 +117,7 @@ returns:
"name": {
"full_name": "author.name",
"mapping": {
"name": { "type": "string", "index_name": "author" }
"name": { "type": "string" }
}
}
}
Expand All @@ -130,7 +127,7 @@ returns:

Note how the response always use the same fields specified in the request as keys.
The `full_name` in every entry contains the full name of the field whose mapping were returned.
This is useful when the request can refer to to multiple fields (like `text` above).
This is useful when the request can refer to to multiple fields.

[float]
=== Other options
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/mapping/fields/all-field.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Here is a sample mapping:
"properties" : {
"first" : {
"properties" : {
"location" : {"type" : "string", "store" : true, "index_name" : "firstLocation"}
"location" : {"type" : "string", "store" : true}
}
},
"last" : {
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/mapping/types/array-type.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ is a sample explicit mapping:
"tweet" : {
"properties" : {
"message" : {"type" : "string"},
"tags" : {"type" : "string", "index_name" : "tag"},
"tags" : {"type" : "string"},
"lists" : {
"properties" : {
"name" : {"type" : "string"},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,13 @@ setup:
t2:
type: string
obj:
path: just_name
properties:
t1:
type: string
i_t1:
type: string
index_name: t1
i_t3:
type: string
index_name: t3

- do:
indices.create:
Expand All @@ -35,16 +32,13 @@ setup:
t2:
type: string
obj:
path: just_name
properties:
t1:
type: string
i_t1:
type: string
index_name: t1
i_t3:
type: string
index_name: t3

- do:
cluster.health:
Expand All @@ -71,12 +65,9 @@ setup:
index: test_index
field: "t*"

# i_t1 matches the pattern using it's index name, but t1 already means a full name
# of a field and thus takes precedence.
- match: {test_index.mappings.test_type.t1.full_name: t1 }
- match: {test_index.mappings.test_type.t2.full_name: t2 }
- match: {test_index.mappings.test_type.t3.full_name: obj.i_t3 }
- length: {test_index.mappings.test_type: 3}
- length: {test_index.mappings.test_type: 2}

---
"Get field mapping with *t1 for fields":
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
import org.apache.lucene.search.*;
import org.apache.lucene.util.BytesRef;
import org.elasticsearch.ElasticsearchIllegalArgumentException;
import org.elasticsearch.Version;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.collect.ImmutableOpenMap;
import org.elasticsearch.common.lucene.BytesRefs;
Expand Down Expand Up @@ -259,16 +260,26 @@ public T copyTo(CopyTo copyTo) {
return builder;
}

public Names buildNames(BuilderContext context) {
return new Names(name, buildIndexName(context), indexName == null ? name : indexName, buildFullName(context), context.path().sourcePath());
protected Names buildNames(BuilderContext context) {
return new Names(name, buildIndexName(context), buildIndexNameClean(context), buildFullName(context), context.path().sourcePath());
}

public String buildIndexName(BuilderContext context) {
protected String buildIndexName(BuilderContext context) {
if (context.indexCreatedVersion().onOrAfter(Version.V_2_0_0)) {
return buildFullName(context);
}
String actualIndexName = indexName == null ? name : indexName;
return context.path().pathAsText(actualIndexName);
}

protected String buildIndexNameClean(BuilderContext context) {
if (context.indexCreatedVersion().onOrAfter(Version.V_2_0_0)) {
return buildFullName(context);
}
return indexName == null ? name : indexName;
}

public String buildFullName(BuilderContext context) {
protected String buildFullName(BuilderContext context) {
return context.path().fullPathAsText(name);
}
}
Expand All @@ -287,6 +298,7 @@ public String buildFullName(BuilderContext context) {
protected FieldDataType fieldDataType;
protected final MultiFields multiFields;
protected CopyTo copyTo;
protected final boolean writePre20Metadata;

protected AbstractFieldMapper(Names names, float boost, FieldType fieldType, Boolean docValues, NamedAnalyzer indexAnalyzer,
NamedAnalyzer searchAnalyzer, PostingsFormatProvider postingsFormat,
Expand Down Expand Up @@ -340,6 +352,9 @@ protected AbstractFieldMapper(Names names, float boost, FieldType fieldType, Boo
}
this.multiFields = multiFields;
this.copyTo = copyTo;
// the short circuit check to EMPTY here is necessary because some built in fields pass EMPTY for simplified ctors
this.writePre20Metadata = indexSettings != null && indexSettings.equals(ImmutableSettings.EMPTY) == false &&
Version.indexCreated(indexSettings).before(Version.V_2_0_0);
}

@Nullable
Expand Down Expand Up @@ -690,7 +705,7 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
protected void doXContentBody(XContentBuilder builder, boolean includeDefaults, Params params) throws IOException {

builder.field("type", contentType());
if (includeDefaults || !names.name().equals(names.indexNameClean())) {
if (writePre20Metadata && (includeDefaults || !names.name().equals(names.indexNameClean()))) {
builder.field("index_name", names.indexNameClean());
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -188,10 +188,8 @@ public static class TypeParser implements Mapper.TypeParser {
} else if (Fields.MAX_INPUT_LENGTH.match(fieldName)) {
builder.maxInputLength(Integer.parseInt(fieldNode.toString()));
iterator.remove();
} else if ("fields".equals(fieldName) || "path".equals(fieldName)) {
if (parseMultiField(builder, name, parserContext, fieldName, fieldNode)) {
iterator.remove();
}
} else if (parseMultiField(builder, name, parserContext, fieldName, fieldNode)) {
iterator.remove();
} else if (fieldName.equals(Fields.CONTEXT)) {
builder.contextMapping(ContextBuilder.loadMappings(fieldNode, parserContext.indexVersionCreated()));
iterator.remove();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ public class TypeParsers {
Map.Entry<String, Object> entry = iterator.next();
String fieldName = Strings.toUnderscoreCase(entry.getKey());
Object fieldNode = entry.getValue();
if (fieldName.equals("path")) {
if (fieldName.equals("path") && parserContext.indexVersionCreated().before(Version.V_2_0_0)) {
pathType = parsePathType(name, fieldNode.toString());
iterator.remove();
} else if (fieldName.equals("fields")) {
Expand Down Expand Up @@ -188,7 +188,7 @@ public static void parseField(AbstractFieldMapper.Builder builder, String name,
Map.Entry<String, Object> entry = iterator.next();
final String propName = Strings.toUnderscoreCase(entry.getKey());
final Object propNode = entry.getValue();
if (propName.equals("index_name")) {
if (propName.equals("index_name") && parserContext.indexVersionCreated().before(Version.V_2_0_0)) {
builder.indexName(propNode.toString());
iterator.remove();
} else if (propName.equals("store")) {
Expand Down Expand Up @@ -303,7 +303,7 @@ public static void parseField(AbstractFieldMapper.Builder builder, String name,
}

public static boolean parseMultiField(AbstractFieldMapper.Builder builder, String name, Mapper.TypeParser.ParserContext parserContext, String propName, Object propNode) {
if (propName.equals("path")) {
if (propName.equals("path") && parserContext.indexVersionCreated().before(Version.V_2_0_0)) {
builder.multiFieldPathType(parsePathType(name, propNode.toString()));
return true;
} else if (propName.equals("fields")) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
import org.apache.lucene.util.BytesRef;
import org.elasticsearch.ElasticsearchIllegalArgumentException;
import org.elasticsearch.ElasticsearchIllegalStateException;
import org.elasticsearch.Version;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.geo.GeoDistance;
Expand Down Expand Up @@ -211,7 +212,7 @@ public static class TypeParser implements Mapper.TypeParser {
Map.Entry<String, Object> entry = iterator.next();
String fieldName = Strings.toUnderscoreCase(entry.getKey());
Object fieldNode = entry.getValue();
if (fieldName.equals("path")) {
if (fieldName.equals("path") && parserContext.indexVersionCreated().before(Version.V_2_0_0)) {
builder.multiFieldPathType(parsePathType(name, fieldNode.toString()));
iterator.remove();
} else if (fieldName.equals("lat_lon")) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
import org.apache.lucene.util.BytesRef;
import org.elasticsearch.ElasticsearchIllegalStateException;
import org.elasticsearch.ElasticsearchParseException;
import org.elasticsearch.Version;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.collect.CopyOnWriteHashMap;
Expand Down Expand Up @@ -217,7 +218,7 @@ public Mapper.Builder parse(String name, Map<String, Object> node, ParserContext
Map.Entry<String, Object> entry = iterator.next();
String fieldName = Strings.toUnderscoreCase(entry.getKey());
Object fieldNode = entry.getValue();
if (parseObjectOrDocumentTypeProperties(fieldName, fieldNode, parserContext, builder) || parseObjectProperties(name, fieldName, fieldNode, builder)) {
if (parseObjectOrDocumentTypeProperties(fieldName, fieldNode, parserContext, builder) || parseObjectProperties(name, fieldName, fieldNode, parserContext, builder)) {
iterator.remove();
}
}
Expand Down Expand Up @@ -252,8 +253,8 @@ protected static boolean parseObjectOrDocumentTypeProperties(String fieldName, O
return false;
}

protected static boolean parseObjectProperties(String name, String fieldName, Object fieldNode, ObjectMapper.Builder builder) {
if (fieldName.equals("path")) {
protected static boolean parseObjectProperties(String name, String fieldName, Object fieldNode, ParserContext parserContext, ObjectMapper.Builder builder) {
if (fieldName.equals("path") && parserContext.indexVersionCreated().before(Version.V_2_0_0)) {
builder.pathType(parsePathType(name, fieldNode.toString()));
return true;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
import org.apache.lucene.search.similarities.Similarity;
import org.apache.lucene.util.Bits;
import org.elasticsearch.Version;
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.ParseField;
import org.elasticsearch.common.lucene.HashedBytesRef;
Expand All @@ -40,6 +41,7 @@
import org.elasticsearch.common.lucene.search.Queries;
import org.elasticsearch.common.lucene.search.ResolvableFilter;
import org.elasticsearch.common.settings.ImmutableSettings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.index.Index;
import org.elasticsearch.index.analysis.AnalysisService;
Expand Down Expand Up @@ -413,7 +415,9 @@ private MapperService.SmartNameFieldMappers failIfFieldMappingNotFound(String na
return fieldMapping;
} else if (mapUnmappedFieldAsString){
StringFieldMapper.Builder builder = MapperBuilders.stringField(name);
StringFieldMapper stringFieldMapper = builder.build(new Mapper.BuilderContext(ImmutableSettings.EMPTY, new ContentPath(1)));
// it would be better to pass the real index settings, but they are not easily accessible from here...
Settings settings = ImmutableSettings.builder().put(IndexMetaData.SETTING_VERSION_CREATED, indexQueryParser.getIndexCreatedVersion()).build();
StringFieldMapper stringFieldMapper = builder.build(new Mapper.BuilderContext(settings, new ContentPath(1)));
return new MapperService.SmartNameFieldMappers(mapperService(), new FieldMappers(stringFieldMapper), null, false);
} else {
Version indexCreatedVersion = indexQueryParser.getIndexCreatedVersion();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,7 @@
"properties":{
"location":{
"type":"string",
"store":"yes",
"index_name":"firstLocation"
"store":"yes"
}
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,7 @@
"properties":{
"location":{
"type":"string",
"store":"yes",
"index_name":"firstLocation"
"store":"yes"
}
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,7 @@
"properties":{
"location":{
"type":"string",
"store":"yes",
"index_name":"firstLocation"
"store":"yes"
}
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,7 @@
"properties":{
"location":{
"type":"string",
"store":"yes",
"index_name":"firstLocation"
"store":"yes"
}
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,7 @@
"properties":{
"location":{
"type":"string",
"store":"yes",
"index_name":"firstLocation"
"store":"yes"
}
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,7 @@
"properties":{
"location":{
"type":"string",
"store":"yes",
"index_name":"firstLocation"
"store":"yes"
}
}
},
Expand Down
Loading