You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At the moment, header_extract only returns data for attributes that it finds. If an attribute is missing, it's just not part of the data["body"]["properties"] dictionary. This seems to be the common behavior among input plugins.
If there are variations in attributes from one file to the next, the data payload will vary. This would be problematic for example with a CSV output plugin, where we'd expect the same columns on every row.
One option is to modify all input plugins to return None for missing attributes. Another would be to have a more elaborate output mechanism, with three steps:
backend.open : initialize backend
backend.export : export data payload
backend.close : finalize backend (write to disk)
The close method could be used to aggregate all properties and fill missing values.
The text was updated successfully, but these errors were encountered:
At the moment,
header_extract
only returns data for attributes that it finds. If an attribute is missing, it's just not part of thedata["body"]["properties"]
dictionary. This seems to be the common behavior among input plugins.If there are variations in attributes from one file to the next, the data payload will vary. This would be problematic for example with a CSV output plugin, where we'd expect the same columns on every row.
One option is to modify all input plugins to return None for missing attributes. Another would be to have a more elaborate output mechanism, with three steps:
The
close
method could be used to aggregate all properties and fill missing values.The text was updated successfully, but these errors were encountered: