Support ignoreRows for TabularResource

[roll]

Overview

Resource specification is created to describe concrete data source with metadata. When we deal with concrete real world data sources there could be some corner case like commented rows or blank rows on top etc. A publisher needs an ability to share this information with implementations.

Example

https://github.com/frictionlessdata/ADB-User-Study/blob/master/metadata.tsv

It's a valid resource (checked by goodtables) except row 2 and 3 which are comments and can't be removed because it's vital metadata for this publisher tools.

Proposal

Introduce ignoreRows (or skipRows or informationalRows or ?) attribute for TabularResource specification. This attribute MUST be an array of integers and strings where:

• numbers mean row number to ignore the row
• strings mean row first characters to match to ignore the row

Example

ignoreRows = [1, 2, "#","//"]

Related

Headers is another example where publisher could be in need of more granular control over data source rows - #326

References

• initial discussion - Support skip rows in goodtables.yml goodtables.io#75

[pwalsh]

closely related to #326

[rufuspollock]

@roll i'm super cautious about this kind of stuff as it is a place where "ETL" logic starts to bleed into the spec and that's a slippery slope. If you delete rows, what about columns, what about transforms etc etc.

Thus, my sense is that ETL stuff like this should not go into the spec for now - at most it should be in patterns and even there i'm cautious.

PS: i am willing to consider #326 because it is so common and it is about presence of a header row.

[roll]

@rufuspollock
@pwalsh has said the same but there is a very common real world problem and it needs some help from specs (may be patterns?). It's only datapackage problem - on other levels implementations could use own options but datapackage encapsulates all knowledge about data sources (and that's the thing - data containerization) so we need some way to allow this information injection (cc @danfowler)

[rufuspollock]

AGREED with @pwalsh: this should go to "Best Practice" rather than spec for now.