Add residual errors to readme.md

[Stephen-Gates]

Desired Behaviour

If publishing invalid data is allowed, add residual errors to readme.md (or to a .json or .csv depending on frictionlessdata/datapackage#364)

To discuss at next sprint planning meeting.

[Stephen-Gates]

@louisjasek @mattRedBox do you have a view on this feature?

At the end of a Validation step, I'm thinking append the errors to the end of the README.md file

### Known Data Errors

This data is published with the following data errors:

Error message 1
...
Error message n

I'm concerned that:

• we don't know if the data was changed but not validated before the export
• that people may not look at the provenance as the export currently fails if no provenance is entered

[louisjasek]

Yeh that's a good point. I guess what is worse for the user?

• correct data with potentially incorrect error reporting
• potentially incorrect data without error reporting

@mattRedBox as a developer, what would you least prefer?

[Stephen-Gates]

See #388 for suggestion giving user control of adding error messages to readme

[ghost]

Hi
@Stephen-Gates , @louisjasek
For residual errors:

• do users have to have these published with the the metadata?
  • if it's not compulsory, then we should just allow the user to choose:
    • if they wish to validate (prompt just before the publishing)
    • if they wish to add the errors to the metadata.
  • if it is compulsory, then we should automatically validate right before publishing and add these errors silently to the metadata.
• compulsory or not compulsory could also be an option in settings/preferences later

I'm not sure I understand why a person needs to look at the provenance report? I guess just making it clear in the how-tos that whatever the process is, is enough. Or we could somehow track whether the provenance data is simply error collation and remind the user that they haven't added their own data - and check that this is OK before export/publish.

[Stephen-Gates]

I’ve suggested that the user has control in #388.

If we add automatically then the error that currently shows when no provenance information is present will not appear. I like this error as it prompts users to think about this.

I guess at some stage we could get fancy and track if data changes after writing errors to readme and warn of potentially incorrect provenance information

[ghost]

Hi @Stephen-Gates
There's possibly a way to keep the data added by user and errors separate. We may be able to show the errors as a read-only display, but keep the logic for user entry separate - I guess it's more about the business logic -> is a provenance file with only errors in it, valid? That should determine whether the user still gets prompted.

[Stephen-Gates]

Makes sense. Protect errors from being changed. Allow user to remove block of errors. Overwrite errors on next "write errors to provenance" command.

[Stephen-Gates]

Invoke "write errors to provenance" command from bottom panel or pop-out error window after error count message, e.g. error count message, button with tooltip

3 error(s) found in this validation.
Write errors to Provenance Information

Rough but that's the general idea

[Stephen-Gates]

@mattredbox can this be closed? (#618 and #615 raised separately)

[ghost]

Hi @Stephen-Gates
I believe it works ok from the main window.
I'm not sure this quite works correctly from the separate Errors window, so either keep open or open a separate issue for this.