Pop-out error message window to deal with a large number of reported errors

[Stephen-Gates]

Explore options to deal with a large number of reported errors.

Options proposed include:

• limit number of errors reported
• display errors in a separate window instead of the bottom panel
• organise errors in some way - tabs, tree view
• display error as comment linked to the cell - https://docs.handsontable.com/0.35.0/demo-comments_.html#example

What option is best or do you have another?

[Stephen-Gates]

Here's an idea...

Display error messages in a table that you can sort and filter (perhaps find)

row number	column name	error type	error message
2	postcode	7	value does meet the required constraint

Perhaps using https://handsontable.com/examples?filters&headers&sorting&dropdown-menu&csv-export

[louisjasek]

Whatever we do as @Stephen-Gates said on our call just now it should support people fixing the error in data curator as well as getting a list to fix the errors in the source system.

I think this rules out limiting the reported errors.

@mattRedBox do any of these stand out as easy to implement?
@Stephen-Gates do you have an opinion regarding what will be most usable for users trying to fix in data curator and also fix in source?

I'm keen to find a balance of the above - let's try close this off today/tomorrow so you can get on with development.

[Stephen-Gates]

If a user can't change the data in Data Curator then, either:

• publish with errors Add residual errors to readme.md #393
• don't publish and correct errors in source system

If a user can change the data in Data Curator then:

• highlight and locate errors Highlight validation errors #275
• show and organise errors to help solve in "batches" but how?

Showing and organising errors

Errors are currently displayed in a fixed bottom panel - this limits the number displayed and the ability to organise. Feedback I've had from TMR is solving errors a column at a time is their preference.

Options:

1. use left side panel for display (my preference as it could match right column property panel with next/previous column arrows - positions us well to do Validate column #276)
2. resizable bottom panel (I believe @mattRedBox said this was hard)
3. fixed bottom panel but with tabs (not sure how to group errors within tabs and would need to make panel larger)
4. open a separate window (like opening keyboard shortcuts - easy to open a new window but I want to avoid the user doing windows management)

@mattRedBox is it easy to mock up option 1 to help decide?

Other suggestions welcome. @TacoSandwich @grhammo @abhiravbansal @albreez

[ghost]

@Stephen-Gates @louisjasek
Stephen option 4, plus opening in a browser as well - see below.
I think the issue we have now with re-using sidepanels, bottom panels is that there may not be enough room to view a list like this.
Yes we could look at a resizable panel - but the css is already pretty complicated now - it's possible we could do something to allow user to pull the tab out of Data Curator, dragging it into its own window, but simpler to just keep the bottom panel there and allow user to open same information as a Data Curator window, and/or a file in a browser:

I understand @Stephen-Gates what you mean by windows management, but for development the easiest way to start is simply:

1. • keep status list at bottom
2. • enable user to also open in a separate window
3. • enable user to open as file in a browser
     @louisjasek All 3 options, here, are super easy to do

The bit that requires a bit more work is the format:
I think putting yet more info into a panel risks the view not being enough for adequate display in the same way that the bottom panel can have too many errors
Option 3 gives the separation from the application: the user is free to explore the file on the filesystem if they wish, but Data Curator would open it as an external link in the default browser.

Once you think that having all 3 is OK,
then yes I can poc the format to see how simple/difficult it is to render as a sortable, but readonly table, etc - not sure if I'd still use handsontable for this or not yet.

The final bit is then deciding how to trigger these extra views - menus, double-clicking in the status panel at bottom, something else?

[Stephen-Gates]

Thanks @mattRedBox. Some questions/thoughts:

• would hyperlink from error to data table work when viewing errors in different window?
• I think if we supported sorting and filtering on the error message table columns, it would negate the need for Validate column #276 and allow users to:
  • focus on errors in a row
  • focus on errors in a column
  • focus on a type error

Main UI could be:

• After validate, errors shown in bottom panel
• bottom panel has close and pop-out buttons
• bottom panel closes on selecting pop-out error messages button
• pop-out button icon
  https://fontawesome.com/icons/external-link?style=light

Pop-out error message "window" UI:

• resizable pop-out error messages window shows sortable/filterable table of error messages
• has close button
• not sure if it has normal windows management (minimise, maximise, close) or is frameless window
• does pop-out window close automatically when validate is re-run or are the contents refreshed and bottom panel refreshed/not shown?
• before error table show:
  • {count} errors detected in last validation
  • button to write error messages to README.md Add residual errors to readme.md #393

https://fontawesome.com/icons/file-check?style=light

Noting that filtering is a paid Handsontables Pro feature, and I'm not a fan of the UI in the column header, I'm open to exploring simpler implementations

Font Awesome 5 external-link

New in Font Awesome 5 the re-designed solid style icon external-link

Font Awesome 5 file-check

New in Font Awesome 5 the re-designed solid style icon file-check

[Stephen-Gates]

/remind me to add acceptance test in 1 weeks time

[reminders]

@Stephen-Gates set a reminder for Mar 23rd 2018

[Stephen-Gates]

Draft Acceptance Test https://relishapp.com/odi-australia/data-curator/docs/tools/validate-table

Feedback welcome @louisjasek @mattRedBox

Validate Table - Tools - Data-curator - ODI Australian Network - Relish

Relish helps your team get the most from Behaviour Driven Development. Publish, browse, search, and organize your Cucumber features on the web.

[ghost]

Hi @Stephen-Gates
Because frictionless collects by row and fields, not columns, I think #276 was always going to be a painful addition - it can be done, but not simply (also because some errors returned only have a row, and no actual column) - so any other way to organise the data with the same result would be better.

[ghost]

Hi @Stephen-Gates
would hyperlink from error to data table work when viewing errors in different window?
With some extra code, Yes. windows can communicate through the main process, but can also borrow the main process libraries to communicate with each other directly.

[Stephen-Gates]

Perhaps instead of https://fontawesome.com/icons/file-check?style=light I create a variation with a cross instead of a tick, given we’re reporting errors

[Stephen-Gates]

Some options

https://fontawesome.com/icons/file-times?style=light
https://fontawesome.com/icons/file-exclamation?style=light
https://github.com/ODIQueensland/data-curator/blob/develop/static/img/validation-results.svg (my vote)

[Stephen-Gates]

The Vue Good Table Column Filters demo looks good:

• sort column
• filter column
• no pagination
• striped table

[reminders]

👋 @Stephen-Gates, add acceptance test time

[Stephen-Gates]

Acceptance Test on Cucumber Pro. Have a read or login to comment.

[ghost]

Pop-out with modification for collapsing status bar while error window open and tab switching is in. Opening file with errors won't in this release.

[Stephen-Gates]

@mattRedBox I think this can be closed and a separate issue opened for

Opening file with errors won't in this release.

In fact I'm not 100% sure what you mean by this - export the validation errors as a csv?

Could you write the issue and close this?

[ghost]

Hi @Stephen-Gates
My understanding from Paul's comments were that they would also like a separate file written with the errors, so that a user has set of errors that they can deal with away from Data Curator if need be.
For Data Curator, the app could open this file in an external browser so that the user has feedback that the IO write has occurred. But yep can do.

[Stephen-Gates]

Now #281