-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No test left behind: stop skipping tests #26088
Labels
Milestone
Comments
We should also consider when and when not to use Skip. |
We should consider when and when not to use Task.CompletedTask. |
ajcvickers
added a commit
that referenced
this issue
Oct 17, 2021
Part of #26088 and #17235 Ideas for better catching behavior changes in the product code. Specifically: - Detect when a negative case stops failing - Detect when a negative case starts failing in a different way Fundamental approach: don't skip tests. In NorthwindAggregateOperatorsQueryTests, we had: - Negative cases that were no longer failing - Negative cases that were skipped for all providers, but worked on some. For example: - Failed on relational, but passed on in-memory - Failed on relational, but passed on Cosmos - Failed on SQL Server, but passed on SQLite - Negative cases that failed in different ways on different providers Specifics: - If a test throws, catch the exception - Were feasible, also validate the exception message or error number - Always call base where possible, rather than repeating the query in an overriden test - Add a standard comment where we have a bug or enhancement tracking the issue. For example: - `// Contains over subquery. Issue #17246.` - Always have an `AssertSql` call in Cosmos and SQL Server tests - Where we expect a provider-specific class to verify SQL, then add a test that checks all test methods are overriden.
Merged
ajcvickers
added a commit
that referenced
this issue
Nov 7, 2021
ajcvickers
added a commit
that referenced
this issue
Nov 18, 2021
ajcvickers
changed the title
Call base in skipped tests
No test left behind: stop skipping tests
Nov 20, 2021
ajcvickers
added a commit
that referenced
this issue
Dec 12, 2021
ajcvickers
added a commit
that referenced
this issue
Dec 12, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Ideas for better catching behavior changes in the product code. Specifically:
Fundamental approach: don't skip tests.
In NorthwindAggregateOperatorsQueryTests, we had:
Specifics:
// Contains over subquery. Issue #17246.
AssertSql
call in Cosmos and SQL Server testsThe text was updated successfully, but these errors were encountered: