Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: option to deal with missing values #45

Open
huard opened this issue Jun 23, 2022 · 0 comments
Open

Feature: option to deal with missing values #45

huard opened this issue Jun 23, 2022 · 0 comments

Comments

@huard
Copy link
Contributor

huard commented Jun 23, 2022

At the moment, header_extract only returns data for attributes that it finds. If an attribute is missing, it's just not part of the data["body"]["properties"] dictionary. This seems to be the common behavior among input plugins.

If there are variations in attributes from one file to the next, the data payload will vary. This would be problematic for example with a CSV output plugin, where we'd expect the same columns on every row.

One option is to modify all input plugins to return None for missing attributes. Another would be to have a more elaborate output mechanism, with three steps:

  • backend.open : initialize backend
  • backend.export : export data payload
  • backend.close : finalize backend (write to disk)

The close method could be used to aggregate all properties and fill missing values.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant