Handle WP NEM12 edge-case with missing MSATSLoadDateTime col #48
Conversation
Western Power NEM12 files don't always include an empty column for MSATSLoadDateTime if it is never present, instead the column can be completely absent. This occurs with files downloaded from their internal portal and causes a ValueError exception to be unnecessarily raised on parse, as the number of columns is less than expected. The AEMO "Meter Data File Format Specification NEM12 & NEM13" states that MSATSLoadDateTime is required if available, but is not mandatory. As such, instead of erroring when the final MSATSLoadDateTime column isn't present, we can proceed and fill it with None with no consequences. Validation for parse_300_row has been modified to ensure that the error is only raised if number of required non-reading fields is less than 6, instead of equals 7. These WP NEM12 files otherwise seem to parse fine with this change.
atyndall
force-pushed
the
western-power-fixes
branch
from
1350d40 to
85c4983
Compare
atyndall
changed the title
|
Hi @atyndall - it looks like one of the tests that @matt-telstra added for his use case is failing as a result of the change. |
|
Hi @aguinane, attached is a sample that works with my changes, but doesn't work with the currently released version. |
|
@matt-telstra I've merged this one in as I'd rather fall on the side of not raising errors then raising if in conflict (can use the strict flag otherwise). 2/3 of your tests still passed but one failed. If you think this test should throw an error still might be worth making use of the strict flag? |
Western Power NEM12 files don't always include an empty column for MSATSLoadDateTime if it is never present, instead the column can be completely absent.
This occurs with files downloaded from their internal portal and causes a ValueError exception to be unnecessarily raised on parse, as the number of columns is less than expected.
The AEMO Meter Data File Format Specification NEM12 & NEM13 states that MSATSLoadDateTime is required if available, but is not mandatory.
As such, instead of erroring when the final MSATSLoadDateTime column isn't present, we can proceed and fill it with None with no consequences.
Validation for parse_300_row has been modified to ensure that the error is only raised if number of required non-reading fields is less than 6, instead of equals 7.
These WP NEM12 files otherwise seem to parse fine with this change.