Schema viewer enhancements.

[emtwo]

This is a prototype of where additional schema info might go. It's a follow-up for some discussion in mozilla#376 (comment)

Essentially each table would have a "?" icon that you can click on to get more information about that table's schema.

In the future, this drawer could have more information in it such as a table or column descriptions or a link to table query samples that are entered on the data source page.

Here are a couple of screenshots:

[emtwo]

Some notes/thoughts:

• @arikfr I know you mentioned using data type icons, I'm wondering if you feel they're still necessary in this table format since now there is more room to write the full type names?

• I only wrote this for postgres but each data source type will need to have custom code added to gather this extra metadata. Perhaps we could start with landing just a couple of them? For now it should just not display the "?" icon if no extra metadata exists.

• For now the data sample query is quite naive, "select * from table limit 1" Perhaps this could be the default with an option for the data source creator to change it to a custom query.

  Aside from these potential improvements, I feel like this PR as it is so far is already useful (maybe just add some tests and a couple more data sources) and we can land it then iterate with further improvements. @arikfr what do you think?

[arikfr]

@arikfr I know you mentioned using data type icons, I'm wondering if you feel they're still necessary in this table format since now there is more room to write the full type names?

I think they are still necessary to show this data in the schema browser, so the user doesn't have to open the drawer every time to see this data.

I only wrote this for postgres but each data source type will need to have custom code added to gather this extra metadata. Perhaps we could start with landing just a couple of them? For now it should just not display the "?" icon if no extra metadata exists.

Yes, it totally makes sense that not all data sources will support this. We can make a generic "SQL data source analyzer" which we can apply to all the SQL based data sources, but still need to support the option that it's not supported by all.

Aside from these potential improvements, I feel like this PR as it is so far is already useful (maybe just add some tests and a couple more data sources) and we can land it then iterate with further improvements. @arikfr what do you think?

Yes, it's definitely doesn't need to be changed much further to land. Even the icons support is something we can later introduce.

The only thing that we should put some thought into is the new data structure the get_schema method will return (related issue: #2553). At first I thought that we shouldn't have a separate metadata object, but just merge it with columns. But now I realize that it has the benefit of not having to change any existing code. Only thing I would consider is to have only one of them. So new data sources will have only metadata while old ones will have columns.

[kocsmy]

Thanks for this, @emtwo

It feels so redundant to have 10-25-50 (?) icon next to each item. How about hiding it just like the >> icon, so we'll only show it on hover. I believe that'd be a more elegant design for this feature.

The icons (type) will be something that can stay there constantly.

[emtwo]

@arikfr, I've made the following updates/changes:

• Use table.metadata when table.columns does not exist. This means that future data sources can have only metadata and this will work.

• I added the method _get_table_sample to BaseQueryRunner as a helper to generalize and it's used by 5 different data sources now to start - presto, mysql, athena, pg, redshift

• Added some tests

• Made the ? button only visible on hover, similar to >>:

[emtwo]

note relevant PR: #2826

[jezdez]

For future reference: #2826

[emtwo]

@arikfr I've updated this PR with some functionality that was discussed during the Redash work week:

• I added two new tables, table_metadata and column_metadata which will store schema information about a table that used to be cached in redis as well as new metadata such as column/table descriptions or examples.

• The refresh_schema celery task that runs periodically now updates schemas stored in these tables. It will never delete a table or column but will only mark the exists field as False and if the table/column re-appears it will mark it as True again.

• Fetching a schema no longer looks it up in the redis cache, but instead looks up whatever data is currently available in the metadata tables.

• The front-end will not display tables or columns with exists marked as False