Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: faster testing with less duplication #460

Merged
merged 1 commit into from
May 12, 2022

Conversation

henryiii
Copy link
Contributor

@henryiii henryiii commented Apr 18, 2022

This replaces the SDist and wheel tests with tests of the file contents of the SDist and wheel. Every file is checked - over-including or forgetting a file will be detected. This also simplifies the path test and runs it on only one matrix element (all OS's) in order to reduce our testing load - currently we fill and block the entire PyPI's quota when running. If something is added that breaks build's ability to run from the path, it should break that matrix element.

With the newly reduced testing load, I've then enabled testing PyPy 3.8 and 3.9 as well.

Copy link
Contributor

@gaborbernat gaborbernat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll need to get the CI to pass but otherwise looks alright.

tox.ini Show resolved Hide resolved
@layday layday linked an issue Apr 20, 2022 that may be closed by this pull request
@henryiii
Copy link
Contributor Author

You'll need to get the CI to pass but otherwise looks alright.

This is broken on main too - pypy + pytest-rerunfailures + Windows blows up. I'll need to play with this and probably report it to PyPy. Maybe we can turn off rerun failure on Windows PyPy...

I didn't realize you could link issues directly in the UI without mentioning them, interesting.

@henryiii henryiii force-pushed the henryiii/tests/faster branch from f6f64e2 to 15afaf7 Compare April 21, 2022 03:58
@gaborbernat
Copy link
Contributor

This is broken on main too - pypy + pytest-rerunfailures + Windows blows up.

Shall we mark it then xfail and report it upstream, when they fix it we can re-enable it 🤔

@henryiii
Copy link
Contributor Author

henryiii commented May 11, 2022

@mattip, any idea why this is INTERNALERROR'ing pytest on all versions of PyPy on Windows? It didn't used to before the last release as far as I can tell. I've been trying to set up to test this off and on for the last three weeks, but it's hard to recreate locally - at least a simple pytest + pypy + windows + pytest-rerunfailures loaded without the INTERNALERROR on a demo test suite. (PyCon and a bunch of other things keep getting in the way, and I specifically have to be at home to access my Windows box).

@mattip
Copy link
Contributor

mattip commented May 11, 2022

It seems on PyPy report.longrepr is a string, and not whatever it should be. Do we know anyone who understands pytest debugging?

@mattip
Copy link
Contributor

mattip commented May 11, 2022

This is broken on main too - pypy + pytest-rerunfailures + Windows blows up

My guess is that any failing test (maybe PyPy only?) triggers some bad path through the report parsing. I submitted #467 to fix the test failures, which were actually tests marked xfail that are passing.

@gaborbernat
Copy link
Contributor

@henryiii I think if you rebase now it should pass 👍

@henryiii henryiii force-pushed the henryiii/tests/faster branch from 14ba3e5 to 22c3320 Compare May 11, 2022 15:28
Copy link
Contributor

@gaborbernat gaborbernat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@henryiii henryiii force-pushed the henryiii/tests/faster branch from 22c3320 to 0ce89ee Compare May 11, 2022 17:20
@layday layday merged commit bebe815 into pypa:main May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Explain why different test suites are needed
4 participants