Add option to not replace "." with "_" in field names

Elasticsearch does not allow "." characters in field names since version 2.0.
Support has been restored since version 5.0.
For compatibility, graylog replaces "." with "_".
However, when Elasticsearch >= 5.0 is used, this is unnecessary.
For instance, Wazuh Indexer from the Wazuh project is forked from
Opensearch 1.3. The character replacement causes issues with Wazuh,
as the Dashboard expects dots as separator in the field name.

This adds the option `replace_dots_in_field_names` to revert this behavior and
allow the use of ".". The replacement is enabled by default for
compatibility with existing graylog configurations.

The extractor configuration in the web interface has been modified to show
a warning when the user inputs a "." in the "Key separator" field.

Closes: Graylog2#4583
Closes: Graylog2#6588
Closes: Graylog2#13043
Closes: Graylog2#14901
Bug: elastic/elasticsearch#19443

graylog2-server/src/main/java/org/graylog2/plugin/Message.java

@@ -19,6 +19,7 @@
 import com.codahale.metrics.Meter;
 import com.eaio.uuid.UUID;
 import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.joschi.jadconfig.Parameter;
 import com.google.common.base.Function;
 import com.google.common.base.Joiner;
 import com.google.common.base.Objects;
@@ -299,6 +300,9 @@ public class Message implements Messages, Indexable {
 
     private static final IdentityHashMap<Class<?>, Integer> classSizes = Maps.newIdentityHashMap();
 
+    @Parameter(value = "replace_dots_in_field_names")
+    private boolean replaceDotsInFieldNames = true;
+
     static {
         classSizes.put(byte.class, 1);
         classSizes.put(Byte.class, 1);
@@ -402,9 +406,10 @@ public Map<String, Object> toElasticSearchObject(ObjectMapper objectMapper, @Non
             }
 
             final Object value = entry.getValue();
-            // Elasticsearch does not allow "." characters in keys since version 2.0.
+            // Elasticsearch does not allow "." characters in keys from versions 2.0 to 5.0 (excluded).
             // See: https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_mapping_changes.html#_field_names_may_not_contain_dots
-            if (key.contains(".")) {
+            // See: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/release-notes-5.0.0.html#enhancement-5.0.0
+            if (key.contains(".") && replaceDotsInFieldNames) {
                 final String newKey = key.replace('.', KEY_REPLACEMENT_CHAR);
 
                 // If the message already contains the transformed key, we skip the field and emit a warning.

graylog2-web-interface/src/components/extractors/extractors_configuration/JSONExtractorConfiguration.jsx

@@ -100,6 +100,12 @@ const JSONExtractorConfiguration = createReactClass({
     return this.state.trying || !this.props.exampleMessage;
   },
 
+  _keySeparatorError(separator) {
+    return separator.includes(".")
+      ? "Warning: Elasticsearch does not allow '.' in field names from version 2.0 up to 5.0 (excluded)"
+      : null
+  },
+
   render() {
     return (
       <div>
@@ -129,6 +135,7 @@ const JSONExtractorConfiguration = createReactClass({
                defaultValue={this.state.configuration.key_separator}
                required
                onChange={this._onChange('key_separator')}
+               error={this._keySeparatorError(this.state.configuration.key_separator)}
                help={<span>What string to use to concatenate different keys of a nested JSON object (only used if <em>not</em> flattened).</span>} />
 
         <Input type="text"

misc/graylog.conf

@@ -406,6 +406,13 @@ allow_leading_wildcard_searches = false
 # should only be enabled after making sure your Elasticsearch cluster has enough memory.
 allow_highlighting = false
 
+# Replace "." characters in keys with "_". Dots are not allowed by Elasticsearch version 2.0 up to 5.0 (excluded).
+# Support have been restored in Elasticsearch 5.0.
+# See: https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_mapping_changes.html#_field_names_may_not_contain_dots
+# See: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/release-notes-5.0.0.html#enhancement-5.0.0
+# Default: true
+#replace_dots_in_field_names = true
+
 # Global timeout for index optimization (force merge) requests.
 # Default: 1h
 #elasticsearch_index_optimization_timeout = 1h