-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add null_value_pattern and null_empty_string option for LabeledTSVParser, CSVParser, TSVParser. #657
Conversation
I think 1. is good and it is good enough for now. I may want to use using 1. and 2. together occasionally. So, alternative idea is to add |
I think this PR is simple enough and good to merge. Any concerns, @repeatedly? |
@frsyuki
I see. Please wait a moment... |
…ser. remove null_value option.
Thank you for advice! I add null_value_pattern and null_empty_string option for LabeledTSVParser. settings example.
null_value_pattern
null_empty_string
|
value = (value == '') ? nil : value | ||
end | ||
if null_value_pattern | ||
value = (value =~ /#{null_value_pattern}/) ? nil : value |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Creating regexp object at each record is high cost.
It should be done in configure method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thank you!
I'll fix it.
Why don't we add these options for super class (ValuesParser)? |
@sonots @repeatedly |
Certainly, in the format like below with MySQL tsv restore
|
Is it official LTSV specification? I thought it is just a custom. |
That's right! javahttps://github.com/making/ltsv4j/blob/master/src/main/java/am/ik/ltsv4j/LTSVParser.java pythonhttps://github.com/hekyou/python-ltsv/blob/master/ltsv/_reader.py |
I'll fix in this way. |
move this option LabeledTSVParser to ValuesParser.
fixed. |
@@ -308,11 +310,16 @@ def configure(conf) | |||
end | |||
|
|||
@time_parser = TimeParser.new(@time_format) | |||
|
|||
if @null_value_pattern | |||
@null_value_pattern = /#{@null_value_pattern}/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regexp.new
is better because users do not need to care of escaping /
with it.
reviewed |
thank you for review. |
Apply this review. thanks. |
added Fluent::StringUtil, filter-grep-plugin and this feature common procesing. |
Looks good to me. Any concerns? > @repeatedly |
Nothing. |
Add null_value_pattern and null_empty_string option for LabeledTSVParser, CSVParser, TSVParser.
Merged! |
@repeatedly @sonots @frsyuki thank you for many advise! 😊 |
Released v0.12.16! |
😊 |
I found that |
@tagomoris Sorry and Thanks. I will check it. |
Changes Unknown when pulling 0ec1c53 on toyama0919:master into ** on fluent:master**. |
Hello.
Cannot express NULL in LTSV.
ltsv library for Ruby blank to NULL.
https://github.com/condor/ltsv/blob/master/lib/ltsv.rb#L97
I use LTSV with apache.
Apache log is If value not exists with hyphen('-').
I think hyphens to null, any of the following methods.
※this PR
settings example.
Which one is good?