Move test files into test subdirectories

[vstinner]

Feature or enhancement

Proposal:

The Python test suite has around 166 files and sub-directories where the name is not clearly associated to a test. For example, it's not obvious to me which test uses talos-2019-0758.pem or coding20731.py.

When possible, I propose to move these files into sub-directories related to their test. Example:

• create Lib/test/test_module/ sub-directory
• move Lib/test/test_module.py to Lib/test/test_module/__init__.py
• move good_getattr.py and bad_getattr*.py scripts to Lib/test/test_module/

Well, I created PR #108293 for this specific example.

Linked PRs

• gh-108303: Remove unused Lib/test/sgml_input.html #108305
• gh-108303: Add Lib/test/test_cppext/ sub-directory #108325
• [3.12] gh-108303: Add Lib/test/test_cppext/ sub-directory (GH-108325) #108328
• [3.11] gh-108303: Add Lib/test/test_cppext/ sub-directory (#108325) #108336
• gh-108303: Move ann_module*.py files to typinganndata/ folder #108354
• gh-108303: Create Lib/test/test_dataclasses/ directory #108978
• gh-108303: Create Lib/test/test_dataclasses/ directory #108979
• gh-108303: Move tokenize-related data to Lib/test/tokenizedata #109265
• gh-108303: Add Lib/test/tokenizedata to TESTSUBDIRS #109314
• gh-108303: Move test_future into its own subdir #109368
• gh-108303: Move all certificates to Lib/test/certdata/ #109489
• gh-108303: Move all math files to Lib/test/mathdata/ #109512
• gh-108303: Fix and move badsyntax_pep3120.py #109513
• gh-108303: Move all inspect test files to test_inspect/ #109607
• [3.11] gh-108303: Remove unused Lib/test/sgml_input.html (GH-108305) #109670
• [3.12] gh-108303: Remove unused Lib/test/sgml_input.html (GH-108305) #109671
• [3.12] gh-108303: Move ann_module*.py files to typinganndata/ folder (GH-108354) #109672
• [3.11] gh-108303: Move ann_module*.py files to typinganndata/ folder (#108354) #109673
• [3.12] gh-108303: Create Lib/test/test_dataclasses/ directory (GH-108978) #109674
• [3.11] gh-108303: Create Lib/test/test_dataclasses/ directory (#108978) #109675
• [3.12] gh-108303: Move tokenize-related data to Lib/test/tokenizedata (GH-109265) #109677
• [3.11] gh-108303: Move tokenize-related data to Lib/test/tokenizedata (GH-109265) #109678
• [3.12] gh-108303: Move test_future into its own test_future_stmt subdir (#109368) #109679
• [3.11] gh-108303: Move test_future into its own test_future_stmt subdir (#109368) #109680
• [3.12] gh-108303: Move all certificates to Lib/test/certdata/ (GH-109489) #109682
• [3.11] gh-108303: Move all certificates to Lib/test/certdata/ (GH-109489) #109683
• gh-108303: Update test_fractions for new Lib/test/mathdata/ #109686
• [3.11] gh-108303: Fix and move badsyntax_pep3120.py (GH-109513) #109724
• [3.11] [3.12] gh-108303: Move all certificates to Lib/test/certdata/ (GH-109489) (GH-109682) #110646
• [3.12] gh-108303: Move all inspect test files to test_inspect/ (GH-109607) #110732
• [3.11] gh-108303: Move all inspect test files to test_inspect/ (GH-109607) #111543
• gh-108303: Move zip and tar archives to Lib/test/archivetestdata #111549
• gh-108303: Move more typing related files to Lib/test/typinganndata #111825
• [3.12] gh-108303: Move more typing related files to Lib/test/typinganndata (GH-111825) #111859
• [3.11] gh-108303: Move more typing related files to Lib/test/typinganndata (GH-111825) #111860
• gh-108303: Move config parser data to Lib/test/configparserdata/ #111879
• gh-108303: Move more files to Lib/test/test_module #111880
• [3.12] gh-108303: Move config parser data to Lib/test/configparserdata/ (gh-111879) #111882
• [3.11] gh-108303: Move config parser data to Lib/test/configparserdata/ (gh-111879) #111883
• [3.12] gh-108303: Move more files to Lib/test/test_module (GH-111880) #111891
• [3.11] gh-108303: Move more files to Lib/test/test_module (GH-111880) #111892
• gh-108303: Install Lib/test/configdata #111899
• [3.11] gh-108303: Install Lib/test/configdata (GH-111899) #111945
• [3.12] gh-108303: Install Lib/test/configdata (GH-111899) #111946
• gh-108303: Move double_const to test_import where it belongs #112108
• gh-108303: Move all doctest related files and tests to Lib/test/test_doctest/ #112109
• gh-108303: Delete imp_dummy test file #112110
• [3.12] gh-108303: Delete imp_dummy test file (GH-112110) #112114
• [3.12] gh-108303: Move double_const to test_import where it belongs (GH-112108) #112976
• [3.11] gh-108303: Move double_const to test_import where it belongs (GH-112108) #112977
• [3.12] gh-108303: Move all doctest related files and tests to Lib/test/test_doctest/ (GH-112109) #114254
• [3.11] gh-108303: Move all doctest related files and tests to Lib/test/test_doctest/ (GH-112109) #114313
• gh-108303: Move .whl test files to Lib/test/whldata/ #114343
• gh-108303: Move all XML-related test files to test_xml #114344
• gh-108303: Remove the non-test Lib/test/reperf.py #114356
• gh-108303: Move smtpd to test.support #114368
• gh-108303: Remove Lib/test/shadowed_super.py #114372
• [3.12] gh-108303: Move smtpd to test.support (GH-114368) #114427
• [3.12] gh-108303: Remove Lib/test/shadowed_super.py (GH-114372) #114433
• [3.12] gh-108303: Move .whl test files to Lib/test/wheeldata/ (GH-114343) #114488
• gh-108303: Move all pydoc related files to test_pydoc #114506
• gh-108303: Move Lib/test/sortperf.py to Tools/scripts #114687
• [3.11] gh-108303: Move all pydoc related files to test_pydoc (GH-114506) #115501
• [3.12] gh-108303: Move all pydoc related files to test_pydoc (GH-114506) #115502
• [3.12] gh-108303: Move Lib/test/sortperf.py to Tools/scripts (GH-114687) #115625
• [3.11] gh-108303: Move Lib/test/sortperf.py to Tools/scripts (GH-114687) #115626

[vstinner]

Files (161):

allsans.pem
ann_module2.py
ann_module3.py
ann_module4.py
ann_module5.py
ann_module6.py
ann_module7.py
ann_module8.py
ann_module.py
audiodata/
audiotests.py
audit-tests.py
autotest.py
badcert.pem
bad_coding2.py
bad_coding.py
badkey.pem
badsyntax_3131.py
badsyntax_future10.py
badsyntax_future3.py
badsyntax_future4.py
badsyntax_future5.py
badsyntax_future6.py
badsyntax_future7.py
badsyntax_future8.py
badsyntax_future9.py
badsyntax_pep3120.py
bisect_cmd.py*
capath/
cfgparser.1
cfgparser.2
cfgparser.3
cjkencodings/
clinic.test.c
cmath_testcases.txt
coding20731.py
crashers/
curses_tests.py*
data/
dataclass_module_1.py
dataclass_module_1_str.py
dataclass_module_2.py
dataclass_module_2_str.py
dataclass_textanno.py
datetimetester.py
decimaltestdata/
dis_module.py
double_const.py
dtracedata/
empty.vbs
encoded_modules/
exception_hierarchy.txt
ffdh3072.pem
final_a.py
final_b.py
floating_points.txt
fork_wait.py
formatfloat_testcases.txt
future_test1.py
future_test2.py
gdb_sample.py
idnsans.pem
ieee754.txt
imp_dummy.py
__init__.py
inspect_fodder2.py
inspect_fodder.py
inspect_stock_annotations.py
inspect_stringized_annotations_2.py
inspect_stringized_annotations.py
keycert2.pem
keycert3.pem
keycert4.pem
keycertecc.pem
keycert.passwd.pem
keycert.pem
leakers/
levenshtein_examples.json
libregrtest/
list_tests.py
lock_tests.py
__main__.py
make_ssl_certs.py
mapping_tests.py
math_testcases.txt
memory_watchdog.py
mime.types
mock_socket.py
mod_generics_cache.py
mp_fork_bomb.py
mp_preload.py
multibytecodec_support.py
nokia.pem
nosan.pem
nullbytecert.pem
nullcert.pem
pickletester.py
profilee.py
pstats.pck
pycacert.pem
__pycache__/
pycakey.pem
pyclbr_input.py
pydocfodder.py
pydoc_mod.py
pythoninfo.py
randv2_32.pck
randv2_64.pck
randv3.pck
recursion.tar
regrtest.py*
relimport.py
reperf.py
re_tests.py*
revocation.crl
sample_doctest.py
secp384r1.pem
selfsigned_pythontestdotnet.pem
seq_tests.py
setup_testcppext.py
setuptools-67.6.1-py3-none-any.whl
shadowed_super.py
signalinterproctester.py
smtpd.py*
sortperf.py
ssl_cert.pem
ssl_key.passwd.pem
ssl_key.pem
ssl_servers.py
ssltests.py
string_tests.py
subprocessdata/
support/
talos-2019-0758.pem
testcodec.py
_testcppext.cpp
test_difflib_expect.html
test_doctest2.txt
test_doctest3.txt
test_doctest4.txt
test_doctest.txt
testtar.tar
testtar.tar.xz
tf_inherit_check.py
time_hashlib.py
tkinterdata/
tokenize_tests-latin1-coding-cookie-and-utf8-bom-sig.txt
tokenize_tests-no-coding-cookie-and-utf8-bom-sig-only.txt
tokenize_tests.txt
tokenize_tests-utf8-coding-cookie-and-no-utf8-bom-sig.txt
tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txt
tracedmodules/
_typed_dict_helper.py
typinganndata/
wheel-0.40.0-py3-none-any.whl
win_console_handler.py
xmltestdata/
xmltests.py
zip_cp437_header.zip
zipdir.zip
ziptestdata/

[vstinner]

The alternative is to have a "data" directory, so a different directory name:

• audiodata/
• decimaltestdata/
• dtracedata/
• subprocessdata/
• tkinterdata/
• typinganndata/
• xmltestdata/
• ziptestdata/

[vstinner]

Existing tests as a sub-directory (proposed solution):

• test_asyncio/
• test_capi/
• test_ctypes/
• test_email/
• test_import/
• test_importlib/
• test_json/
• test_lib2to3/
• test_module/
• test_peg_generator/
• test_sqlite3/
• test_tkinter/
• test_tomllib/
• test_tools/
• test_ttk/
• test_unittest/
• test_warnings/
• test_zipfile/
• test_zoneinfo/

[terryjreedy]

I have also noticed and wondered about the melange of misc files. I like packaging the test code and data together in one directory, as in your 'module' example, especially given that we already have test_xyz directories.

[vstinner]

I created PR #108299 for test_crashers.

[serhiy-storchaka]

What with data files used in several different tests?

[vstinner]

What with data files used in several different tests?

For this case, I suggest to keep the current trend of having a "data/" directory, such as audiodata/. I don't think that duplicating files to "isolate" tests is worth it.

The problem with these "data" directories is that when a test file is no longer needed, it's more likely to be forgotten.

[vstinner]

I close the issue. While it's not 100% done, remaining files can reuse this closed issue, or another issue can be created. This issue already has a long list of pull requests and a long history.

Thanks to everybody who helped cleaning Lib/test/, reviewers, and especially @sobolevn who wrote many changes!