Initial pass at switching integration tests to pytest #4691

gshank · 2022-02-07T01:54:38Z

resolves #4690

Description

This pull request is for preliminary review of the state of the integration test conversion to pytest.

Checklist

I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change

jtcohen6

This is really exciting!! Lots of comments as I read through the draft implementation.

As next steps, I'd be excited to:

Try porting the two converted integration tests to an adapter plugin, and see how we'd need to reimplement fixtures/methods/etc
Think about what other modular utilities we'd want, in a similar vein to get_manifest. I think @JCZuurmond's work on pytest-dbt-core could be generative here, especially macro_generator to mock macro input + output. Could that do for "execution" code what get_manifest does for "parsing" code?

tests/tables.py

tests/util.py

tests/functional/conftest.py

tests/functional/simple_copy/test_mixed_case_db.py

tests/functional/simple_copy/test_copy_uppercase.py

tests/functional/simple_reference/test_invalid_reference.py

tests/functional/simple_reference/test_simple_reference.py

jtcohen6 · 2022-02-07T14:40:23Z

tests/functional/simple_copy/test_simple_copy.py

+        'advanced_incremental.sql': advanced_incremental_sql,
+        'compound_sort.sql': compound_sort_sql,
+        'disabled.sql': disabled_sql,
+        'empty.sql': empty_sql,
+        'get_and_ref.sql': get_and_ref_sql,
+        'incremental.sql': incremental_sql,
+        'interleaved_sort.sql': interleaved_sort_sql,
+        'materialized.sql': materialized_sql,
+        'schema.yml': schema_yml,
+        'view_model.sql': view_model_sql,


Should we aim to consolidate these fixtures with the ones used in other tests? It would require more upfront work, but if we assembled a common set of standard fixtures (rather than recreating highly similar ones over and over again), then:

each test can pick & choose just the ones it needs

adapter plugins could reimplement specific fixtures (as needed) among the standard reused set, and then be able to get most tests "for free," rather than needing to reimplement similar fixtures in similar ways for each test

Ideally I think it would be nice to have some standard projects that can be reused. We have a lot of tests that create projects with only subtle or minimal differences. That can be done a directory basis, using a test directory conftest.py, with only the different parts overridden. We probably could do that for test_simple_copy.py and test_copy_upperbase, and just override the 'models' fixture for test_copy_upperbase. You can also make individual sets of project fixtures with different names and include them in a test as a plugin.

I think it will be easiest to do that after we've converted the tests though, since we'll then be able to more easily see the differences between test projects. The legacy tests with all the files in directories (often multiple) makes that sort of comparison harder.

Though on second thought, if people see the same setup as they're going through converting, there's no reason to wait. "Oh, I saw this project before. Yank! It's now a plugin."

tests/CONVERTING.md

JCZuurmond

Hi, super nice you are upgrading the testing framework! I added some comments, I am still thinking about how I would do this myself, view the comments as suggestions, probably some education is needed before I understand the intent of the changes.

It is great to remove all the dbt subprocess calls as this allows the user to start debugging sessions.

Furthermore, it would be great to have a test suite which can be applied to the core and all the adapters. Especially, if it is written in such a way that adapters can easily extended it (for example with adapter specific settings) whilst reusing the test suite.

Pytest recommends two test layouts: outside or inside the package. The most common set-up is having your tests outside of your package: this makes your package smaller. However, some packages don't do this (for example numpy).

A benefit of having the tests inside your package is that you can run the test suite as a user. This could be helpful for the adapters:

Install dbt-core: pip install dbt-core
Run dbt-core test suite: pytest --pyargs dbt
If then the test suite is set-up to automatically pick up which adapter is installed, you can reuse that test suite for every adapter.

As mentioned in the comments, I don't think fixtures are usually shipped inside the package. If you want to reuse a fixture over different projects, I think the idea is to write a pytest plugin.

core/dbt/tests/tables.py

core/dbt/tests/util.py

core/dbt/tests/fixtures/project.py

tests/functional/basic/test_jaffle_shop.py

core/dbt/tests/fixtures/project.py

Author: Emily Rockman <[email protected]> Date: Thu Feb 10 15:45:47 2022 -0600 route logs to dbt-core/logs instead of each test folder (#4711) commit 507184f Author: Gerda Shank <[email protected]> Date: Thu Feb 10 16:22:20 2022 -0500 Initial pass at switching integration tests to pytest

Add comments, cleanup adapter schema setup

emmyoop

Looks good with the understanding that we'll make changes as needed as we learn more/establish patterns.

gshank requested review from a team as code owners February 7, 2022 01:54

gshank requested review from emmyoop and leahwicz and removed request for a team February 7, 2022 01:54

cla-bot bot added the cla:yes label Feb 7, 2022

gshank removed the request for review from a team February 7, 2022 01:54

gshank marked this pull request as draft February 7, 2022 01:55

gshank requested a review from jtcohen6 February 7, 2022 01:55

jtcohen6 mentioned this pull request Feb 7, 2022

Add testing dbt-labs/spark-utils#22

Merged

jtcohen6 reviewed Feb 7, 2022

View reviewed changes

JCZuurmond reviewed Feb 9, 2022

View reviewed changes

gshank force-pushed the testing_rewrite branch 12 times, most recently from 2c054b2 to f552cfa Compare February 17, 2022 21:18

gshank force-pushed the testing_rewrite branch 6 times, most recently from 63c460b to 6e48fbf Compare February 18, 2022 00:54

gshank force-pushed the testing_rewrite branch 4 times, most recently from 3dcea97 to fc4adda Compare February 18, 2022 03:28

gshank marked this pull request as ready for review February 18, 2022 13:44

gshank requested a review from a team February 18, 2022 13:44

gshank requested a review from a team as a code owner February 18, 2022 13:44

gshank added 2 commits February 19, 2022 18:36

Reorganize dbt.tests.tables. Cleanup adapter handling

30b5707

gshank force-pushed the testing_rewrite branch from 0a3987f to 30b5707 Compare February 19, 2022 23:46

Move run_sql to TestProjInfo and TableComparison.

635ac34

Add comments, cleanup adapter schema setup

gshank requested a review from ChenyuLInx February 20, 2022 22:37

Tweak unique_schema name generation

e72db2b

gshank force-pushed the testing_rewrite branch from 51aad7c to e72db2b Compare February 22, 2022 17:01

gshank added 2 commits February 22, 2022 15:14

Update CHANGELOG.md

5be74ab

Merge branch 'main' into testing_rewrite

b885c03

emmyoop approved these changes Feb 22, 2022

View reviewed changes

gshank merged commit 8fd8dfc into main Feb 22, 2022

gshank deleted the testing_rewrite branch February 22, 2022 20:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial pass at switching integration tests to pytest #4691

Initial pass at switching integration tests to pytest #4691

gshank commented Feb 7, 2022

jtcohen6 left a comment •

edited

Loading

jtcohen6 Feb 7, 2022

gshank Feb 7, 2022

gshank Feb 7, 2022

JCZuurmond left a comment

emmyoop left a comment

Initial pass at switching integration tests to pytest #4691

Initial pass at switching integration tests to pytest #4691

Conversation

gshank commented Feb 7, 2022

Description

Checklist

jtcohen6 left a comment • edited Loading

Choose a reason for hiding this comment

jtcohen6 Feb 7, 2022

Choose a reason for hiding this comment

gshank Feb 7, 2022

Choose a reason for hiding this comment

gshank Feb 7, 2022

Choose a reason for hiding this comment

JCZuurmond left a comment

Choose a reason for hiding this comment

emmyoop left a comment

Choose a reason for hiding this comment

jtcohen6 left a comment •

edited

Loading