feat(cmd/influx/write): add new processing options and enhancements #18779
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sranka
changed the title
sranka
changed the title
sranka
changed the title
GeorgeMac
reviewed
Jul 7, 2020
|
Thanks for addressing my concern. It looks much better. |
glinton
approved these changes
Sep 11, 2020
pkg/csv2lp/csv_annotations.go
Outdated
Comment on lines
119
to
121
| columnLabel := placeholder[2 : len(placeholder)-1] // ${columnLabel} | ||
| if columnLabel == "$" { | ||
| return nil // ${$} is a way to print $ |
There was a problem hiding this comment.
👍 there is no need to do so
pkg/csv2lp/data_conversion.go
Outdated
| func normalizeNumberString(value string, format string, removeFraction bool) string { | ||
| if len(format) == 0 { | ||
| func normalizeNumberString(value string, format string, removeFraction bool) (normalized string, truncated bool) { | ||
| if format == "" { |
There was a problem hiding this comment.
Suggested change
| if format == "" { | |
| if len(format) == 0 { |
the former method is technically faster
| @@ -110,20 +112,20 @@ func normalizeNumberString(value string, format string, removeFraction bool) str | |||
| } | |||
| if c == fractionRune { | |||
| if removeFraction { | |||
| break ForAllCharacters | |||
| return retVal.String(), true | |||
There was a problem hiding this comment.
i assume this new behavior is intended (not looping through values again)?
There was a problem hiding this comment.
it is intended, the function was changed to return also a truncated indicator, which is true only in this case
sranka
force-pushed
the
18744/csv2lp
branch
from
3867c87 to
78fe5c6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #18744 : indicate a loss of precision when parsing long/unsignedLong data types
When parsing
longorunsignedLongvalue from a string value with fraction digits, the whole CSV row fails when in a strict mode, which is turned on using a column dataFormat that starts withstrict, such aslong:strict. A warning is printed when not in a strict mode, sayingline x: column y: '1.2' truncated to '1' to fit into long data type. Package documentation explains strict parsing.Closes #18742 : add the ability to write rejected records to a specific file
There is a new --error-file option in
influx writeThe file then contains rows than cannot be imported because of data quality issues, CSV comments are also written to let the user know of what was wrong, for example:
Closes #18741: add the ability to construct a timestamp from multiple columns
A new
#concatannotation adds a new column that is concatenated from existing columns according to bash-like string interpolation literal with variables referencing existing column labels. For example:#concat,string,fullName,${firstName} ${lastName}