-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
quarkus-logging-gelf: mapper_parsing_exception when include-log-message-parameters is true (default) #15370
Comments
/cc @loicmathieu |
This is not an issue but a limitation of Elasticsearch, when it first encounters a field, if it has no mapping for it, it will create a mapping for this field based on the filed type (by automatically detecting the type). The type of the field cannot be changed in the mapping. This is not something we can fix or workaround, this is how Elasticsearch works. As I understand it, you already ask the upstream library for a way to fix this, and the only way is to tell Elasticsearch which type is your field to avoid the automatic mapping, you can do it in two ways:
I cannot see anything we can do on our side. |
@loicmathieu So how do I specify the type of parameters via |
extension states that it supports graylog, but literally every person who wants to use it with default settings will face the issue and will have to set this parameters. I believe this deserves a mention in the docs at least |
@loicmathieu @Shohou is saying that he can't define the types because the additional field config requires a value and from what I can see, he's right. So if it's possible with Gelf to only define the type for additional fields (I have no idea if it does but looks like that's what Mark is saying in the other issue), we should also allow this and make the value optional. |
I'm not sure configuring I would suggest to disable Another solution would be to only use String on your log message parameter, so using There is still the possibility to create manually the Elasticsearcg mapping, this would be the prefered way to do it from an Elasticssearch perspective. |
But it looks like Mark is saying you can do it with
Not sure I understand this one. Your types need to be consistent across all messages given you need the schema to accept all log messages. |
I definitely don't want to configure type for each message separately (is it even possible?). I only want all my messages getting to graylog. So String type for Setting quarkus.log.handler.gelf.include-log-message-parameters to false is what I do right now. Making all logging parameters String, it's not possible... Will you also change types of parameters everywhere in quarkus? In 3rd party libraries... I found how to create elasticsearch mapping - https://docs.graylog.org/en/3.2/pages/configuration/elasticsearch.html#custom-index-mappings |
Imagine the following snippet:
These two logs will need to have the types of there messages configured. We may need to ask more clarification to Mark.
True and this is the more complex part, that's why I usggest disabling individual message parameter inclusion inside the generated elasticsearch document, because you would always want String in this case as utilmately everything can be converted to a String. |
A note - you shouldn't enable |
@Shohou I created #18069 with some explaination of what's happening and how to fix it. I don't think we can really do something more about it, it's the developer's (or the ops) responsibilty to configure the Elasticsearch index when using a tool that send data to Elasticsearch so a documentation section should be enought to help. |
Yes, I read the docs and it explain the situation and points to further reading, so I think it's OK |
@Shohou thanks for your feedback. |
Fixes quarkusio#15370 WIP Update docs/src/main/asciidoc/centralized-log-management.adoc
Fixes quarkusio#15370 WIP Update docs/src/main/asciidoc/centralized-log-management.adoc (cherry picked from commit 3e89e28)
Fixes quarkusio#15370 WIP Update docs/src/main/asciidoc/centralized-log-management.adoc
Describe the bug
When using quarkus-logging-gelf with graylog I get
This happens when first appearance of message parameter is long, Graylog (or elasticsearch) remembers the type and next messages with string parameters fail indexing process.
Expected behavior
Message indexing does not fail
Actual behavior
I get indexing errors and messages are lost
To Reproduce
try to log message like
log.info("My long types {}, {}, {}, {}, {}", 1, 2, 3, 4, 5);
and then message with strings
log.info("My string types {}, {}, {}, {}, {}", "a", "b", "c", "d", "e");
Setup logging to graylog and check it indexing errors
Environment (please complete the following information):
Additional context
Disabling message parameters fixes the issue
quarkus.log.handler.gelf.include-log-message-parameters=false
I first filled an issue in logstash-gelf - mp911de/logstash-gelf#264
They expect users of their library to set parameter types. Bad decision IMHO. This parameters are generated automatically and type also should be set automatically...
Quarkus also doesn't allow to set types only. If I add
quarkus.log.handler.gelf.additional-field.MessageParam0.type
quarkus also requires to setquarkus.log.handler.gelf.additional-field.MessageParam0.value
The text was updated successfully, but these errors were encountered: