Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSV file delimiter choice #4778

Closed
phaedradg opened this issue Nov 22, 2024 · 4 comments
Closed

CSV file delimiter choice #4778

phaedradg opened this issue Nov 22, 2024 · 4 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@phaedradg
Copy link

It would be interesting to have more choice in which delimiter is being used in the log files. Now it seems to be a comma.
I encountered a problem when trying to visualize a csv log file in an external tool, because the RB (Redundancy Bus) state field contained a string with multiple status messages, separated by commas. This makes a comma-delimited csv file invalid. If I could select a semicolon for instance, this problem would be solved.
The "RB State" field has this value in the erroneous csv file:
"RX3 Failsafe, RX3 Lost Frame, RX3 Connection Lost, RX3 No Signal"
I end up with 3 less data columns than indicated by the header.

@mawzthefinn
Copy link
Collaborator

mawzthefinn commented Nov 22, 2024

By definition any CSV file can only use a comma separator. CSV actually has meaning, standing for Comma Separated Values

Looks like the header needs expansion any time the field has multiple values in one telemetry field, or ETHOS needs to rewrite any delimiters included in the data (ie rewrite commas to semi-colons if there's commas in the data being written to the CSV file)

@noamkrief
Copy link

@mawzthefinn
Perhaps when it originated, the technique matched the acronym.
These days, we don't take the acronym too literally :)

We should separate the fields with "," (double quote comma double quote)

In my line of work we deal with so called CSVs all the time from various data sources. Sometimes they are separated by ||| or ~~~ just to be a more unlikely case those exist as part of the data.

@robthomson
Copy link
Collaborator

Just needs quotes and \ in any field quotes contained withing a field.

Simples

@bsongis-frsky
Copy link
Collaborator

Keeping the simple comma makes our files compliant with all CSV decoders. We have to escape the comma inside fields.

@bsongis-frsky bsongis-frsky added the bug Something isn't working label Nov 25, 2024
@bsongis-frsky bsongis-frsky self-assigned this Nov 25, 2024
@bsongis-frsky bsongis-frsky added this to the 1.6.0 milestone Nov 25, 2024
@bsongis-frsky bsongis-frsky modified the milestones: 1.6.0, 1.6.1 Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants