📝 Sql query tutorial

[chris-s-friedman]

add a tutorial on one method for sourcing data from a postgresql database and moved the tutorial on sourcing from study creator to the same directory as sourcing data from sql.

This is a potential solution for #627

[fiendish]

that's a very clever trick

[nicholasvk]

Very cool! Just want to chime in with some history/background toward developing standards on when it makes sense to use SQL directly. We thought about doing this for the CBTN refresh work which relies on data sources in the D3b warehouse. However, after discussions as a team with Allison/Bailey, we decided that we should write SQL output to Data Tracker via the study creator API for audit purposes and then use the files on data tracker for actual ingest. Database tables will change over time, rows will be added/deleted and/or updated, so having a record of exactly what is ingested for a given run seemed important at the time. A bonus is that Data Tracker makes this history easily accessible to non-ADAPT team members. You can review the code (authored by Meen and Avi) for that here:
https://github.com/d3b-center/d3b-warehouse-kids-first-refresh/blob/main/etl_from_eig_into_warehouse/dumping.py

Is this the best workflow? Are there other solutions to explore for this? Absolutely, just wanted to add to the conversation with history on the CBTN side.

[chris-s-friedman]

@nicholasvk i think those are really good points about

1. being able to audit what data is used for ingest
2. the volatility inherent to using sql as a data source - i.e. rows being added/ deleted/ updated
3. facilitate non-technical folks be able to have eyes on the data being used for ingest.

my thinking proposing a tutorial on how to query sql in an ingest is for two specific circumstances:

1. ingesting information about genomic files from aws scrapes. instead of having ingesters generate aws manifests, we can pull directly from the file_metadata schema in postgres
2. provide a tutorial for external users that use the ingest library but don't have the data tracker but do use databases as source data.

that said - I think I should prepend this tutorial with a note about your three points and how querying directly from sql impacts those points. Perhaps even say " instead of querying sql, you may want to create a static view of your database in a single file"

[nicholasvk]

Updated documentation for SQL considerations looks great!