feat/sql source #185
feat/sql source #185
Conversation
rbiseck3
force-pushed
the
roman/postgres-source
branch
from
f866f95 to
eda6f9e
Compare
| "password": "test", | ||
| } | ||
| with postgres_download_setup(): | ||
| with tempfile.TemporaryDirectory() as tmpdir: |
There was a problem hiding this comment.
Eventually would love to have this configurable to go to TempDirectory or a specified directory like in the .sh ingest tests. Nice to have.
There was a problem hiding this comment.
Why would we want that? This is a cleaner version of that, making sure the data is always cleanup up since for the test, we never need the contents that are written here.
There was a problem hiding this comment.
I've found it valuable to see what is being downloaded from source connectors in the past. I like to look at the .json files from the indexer and the .txt files from the downloader to troubleshoot. I can always just change it from tempdir to a named folder when I need to so not a big deal. But I always tell new connector devs to run the tests and then look at the resulting files as an example of what their "new" connector needs to be doing in the many phases.
| downloader=downloader, | ||
| configs=ValidationConfigs( | ||
| test_id="sqlite", | ||
| expected_num_files=40, |
There was a problem hiding this comment.
This is great. Would love to have another validation called length of list or something like that so that we could indeed check that only car_id and brand came through.
There was a problem hiding this comment.
Nothing is stopping us from adding an additional test after that one, I can add it in here. This just wraps the most common functionality that is shared by all source connector tests.
* implement sqlite version of sql indexer and downloader * add integration test for sqlite source connector * Drop ids from copied filedata * migrate env setup over to integration test folder * add postgres source connector with tests * bump changelog
Description
Update both the sqlite and postgres implementations to support a source connector.