Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Winlogbeat] Output/Kafka Can’t use topics with uppercase letters anymore #18640

Closed
theodiefenthal opened this issue May 19, 2020 · 4 comments · Fixed by #18854
Closed

[Winlogbeat] Output/Kafka Can’t use topics with uppercase letters anymore #18640

theodiefenthal opened this issue May 19, 2020 · 4 comments · Fixed by #18854
Assignees
@urso
Labels
Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@theodiefenthal
Copy link

Hi there,

I just wanted to upgrade my winlogbeat from 7.2.0 to 7.7.0 but had to realize that the kafka output is broken.

In newer versions, the config option topic from output.kafka goes through a constSelectorExpr and is lowercased before being passed to the kafka client. This means that I can't write to a topic of mine anymore as it contains an uppercase letter.

This limits the number of topics that can be written to unnecessarily. As I don't see a reference to that in the CHANELOG, I'd guess that this was introduced accidentally and can thus be qualified as a.

See also https://discuss.elastic.co/t/bug-output-kafka-cant-use-topics-with-uppercase-letters-anymore/233194 for a discussion thread about it.
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 19, 2020
@andresrc andresrc added the Team:Services (Deprecated) Label for the former Integrations-Services team label May 22, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 22, 2020
@theodiefenthal
Copy link
Author

Just tested 7.6.2 and that's still working. So the break was introduced with 7.7.0

@theodiefenthal theodiefenthal mentioned this issue May 25, 2020
Closed
@xeraa
Copy link

xeraa commented May 28, 2020

Confirmed by another user: https://twitter.com/SwiftOnSecurity/status/1266107095369355270

@urso urso self-assigned this May 28, 2020
@urso
Copy link

urso commented May 28, 2020

The error has been introduced in #16081, which ensures that index names are always lower cased. Unfortunately the Kafka topic configuration reuses the Selector code, and the error went unnoticed.

@urso urso mentioned this issue May 29, 2020
Merged
4 tasks
urso pushed a commit that referenced this issue Jun 10, 2020
Make selector string casing configurable (#18854)
28f7aca 
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. 

## Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (#16081. #6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.
@urso urso mentioned this issue Jun 10, 2020
Merged
4 tasks
urso pushed a commit to urso/beats that referenced this issue Jun 10, 2020
Make selector string casing configurable (elastic#18854)
68af9e9 
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.

## Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

(cherry picked from commit 28f7aca)
urso pushed a commit that referenced this issue Jun 11, 2020
Cherry-pick #18854 to 7.x: Make selector string casing configurable (#…
5fa9630 
…19118)

Make selector string casing configurable (#18854)
    
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.
    
    
Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (#16081. #6342).

As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (#18640).

With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.
    
(cherry picked from commit 28f7aca)
@urso urso mentioned this issue Jun 12, 2020
Merged
4 tasks
urso pushed a commit to urso/beats that referenced this issue Jun 12, 2020
Make selector string casing configurable (elastic#18854)
b71e0c1 
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

(cherry picked from commit 28f7aca)
@urso urso mentioned this issue Jun 25, 2020
Merged
4 tasks
urso pushed a commit to urso/beats that referenced this issue Jun 25, 2020
Make selector string casing configurable (elastic#18854)
5365111 
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.

## Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

(cherry picked from commit 28f7aca)
@kaiyan-sheng kaiyan-sheng mentioned this issue Jul 7, 2020
Closed
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this issue Oct 14, 2020
@melchiormoulin
Make selector string casing configurable (elastic#18854)
a176fba 
Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. 

## Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@urso urso

Labels
Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make selector string casing configurable urso/beats
5 participants
@urso @xeraa @andresrc @theodiefenthal @elasticmachine