Support for Databae queries/Views as data sources

[mtsz-thiago]

As it is not always convenient to keep all data sources as text files, csv and/or related formats, i think it would be nice to declare views and queries from databases as project's data sources.

[efiop]

Hi @mtsz-thiago ! Thanks for the request. Could you please elaborate? Maybe share some thoughts on how you see that working in dvc.

[dmpetrov]

@mtsz-thiago DVC usually works when you already extracted data from DB to files and ready for ML phase (which usually consumes raw data).

However, we are thinking about a tighter integration to DBs (see #1577 and #2378) when you can version and control ML phase and have some huck with some control over DB.

If your scenario is not ML, but more analytical and you spend 100% time in DB then dbt (data build tool) might be a better fit for you.

If I understand your question correctly - this issue is a duplicate of #1577. Please let me know if it's not.

[mike-weinberg]

Data governance on S3 is painful, data access on s3 is slow, and s3 has no built in compute. Warehouse environments like Snowflake, Bigquery, and (recently) redshift all offer low cost storage pricing. In many cases a data science project involves performing data prep on a data warehouse, so why not let the database table be an option for a storage backend. this would greatly reduce the amount of glue code required to make DVC work.

DBT is a good tool, but often times it's more overhead than data scientists want. Don't even get me started on airflow for development.

It's becoming increasingly common for companies to implement snapshot strategies of their raw data because storage prices are so cheap on BQ, Snowflake, etc, and so a snapshot timestamp plus a set of tables is a great abstraction that DVC could leverage. Food for thought.