Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically lowercase the index name when custom fields is used to build it #6342

Closed
daohodac opened this issue Feb 9, 2018 · 5 comments · Fixed by #16081
Closed

Automatically lowercase the index name when custom fields is used to build it #6342

daohodac opened this issue Feb 9, 2018 · 5 comments · Fixed by #16081
Labels
bug good first issue Indicates a good issue for first-time contributors libbeat Team:Integrations Label for the Integrations team

Comments

@daohodac
Copy link

daohodac commented Feb 9, 2018

As uppercase is invalid in elasticsearch indices, Beats should either provide a lowercase processor or always ensure index names are lowercased when using custom fields

Use case (filebeat):

A document:

{"HDG": 125.0, "fwot": "FOO", "date": "2018-01-31 23:00:03"}

A beat configuration:

.......
- type: log
  paths:
    - /var/log/myfeed/*.log
  json.keys_under_root: true
......
output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "http"
  username: "elastic"
  password: "changeme"
  index: "custom-%{[fwot]}-%{+yyyy.MM}-v1"
.....
@exekias exekias added enhancement libbeat good first issue Indicates a good issue for first-time contributors labels Feb 9, 2018
@ph
Copy link
Contributor

ph commented Feb 9, 2018

I would prefer explicit lowercase of the field vs magic. @daohodac are you getting bulk insertion error when uppercase are present?

@daohodac
Copy link
Author

@ph well, not an error, but an error log,. Nothing inserted, just fails.
And beat does not have a dead letter, so it only appears on the logs

@kvch
Copy link
Contributor

kvch commented Apr 18, 2019

Hits to get started:
A new processor needs to be added to lowercase field values. Example processor: https://github.com/elastic/beats/blob/master/libbeat/processors/actions/truncate_fields.go

@urso
Copy link

urso commented Apr 18, 2019

I wonder if the processor is the right place. Maybe we should not tamper with event contents itself, but ensure that the strings we extract when creating the index name are lower-case.

@Joginder22
Copy link

+1 facing same issue with ES output when index name is in uppercase. I'ts better to have lowercase keyword in the output to make every index name in lowercase letters only.

@andresrc andresrc added the Team:Integrations Label for the Integrations team label Mar 6, 2020
urso pushed a commit that referenced this issue Jun 10, 2020
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. 

## Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (#16081. #6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.
urso pushed a commit to urso/beats that referenced this issue Jun 10, 2020
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.

## Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

(cherry picked from commit 28f7aca)
urso pushed a commit that referenced this issue Jun 11, 2020
…19118)

Make selector string casing configurable (#18854)
    
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.
    
    
Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (#16081. #6342).

As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (#18640).

With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.
    
(cherry picked from commit 28f7aca)
urso pushed a commit to urso/beats that referenced this issue Jun 12, 2020
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

(cherry picked from commit 28f7aca)
urso pushed a commit to urso/beats that referenced this issue Jun 25, 2020
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.

## Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

(cherry picked from commit 28f7aca)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this issue Oct 14, 2020
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. 

## Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug good first issue Indicates a good issue for first-time contributors libbeat Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants