test_dis should test the dis module, not everything else

[markshannon]

BPO	46760
Nosy	@terryjreedy, @markshannon, @JelleZijlstra, @brandtbucher, @iritkatriel
PRs	bpo-46760: Remove unnecessary offsets from expected out in test.test_dis #31369

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-15.14:17:08.858>
labels = ['type-feature', 'tests']
title = 'test_dis should test the dis module, not everything else'
updated_at = <Date 2022-02-18.21:38:39.147>
user = 'https://github.com/markshannon'

bugs.python.org fields:

activity = <Date 2022-02-18.21:38:39.147>
actor = 'terry.reedy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2022-02-15.14:17:08.858>
creator = 'Mark.Shannon'
dependencies = []
files = []
hgrepos = []
issue_num = 46760
keywords = ['patch']
message_count = 2.0
messages = ['413291', '413514']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'Mark.Shannon', 'JelleZijlstra', 'brandtbucher', 'iritkatriel']
pr_nums = ['31369']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46760'
versions = []

[markshannon]

<rant>

This is getting really annoying.
It takes longer to fix all the heavily coupled and poorly written tests in test_dis than to make the real changes.

Tiny changes in the calling sequence, or reordering CFGs, cause huge diffs in the test_dis module.
No one ever checks these changes, they are just noise.

I've put this under "enhancement" as there is no "wastes a huge amount of time" category.

</rant>

The test_dis should not:

• Contain offsets; they turn one line diffs into 100 line diffs.
• Contain tests for the compiler; they belong elsewhere.
• Contain big strings; write proper tests not just regex matches.
• Tests for Instruction should should not depend on the compiler output; create the bytecode directly.

This is not a new problem, but it does seem to be getting progressively worse.

A lot of the irritation stems from
b39fd0c
although the tests from before that still hardcode offsets.

[terryjreedy]

With IDLE, I have issues with trying to test IDLE code without retesting tkinter, as well as deciding on the proper units of behavior to test.

Some suggestions:

1. Add a docstring to the module with guidelines, after review from a couple of others.
   For instance,I believe you are saying that no test should explicitly call compile. Rather the test writer should call compile and extract the bytecode to copy into a test.

2. Make yourself a module code owner so you get informed and can review. Reviewing would be easier with guidelines to refer to, instead of repeating them.

3. An example issue is that some callables take many types of arguments. If not already, tests that the function can extract the bytecode from various objects should be separate from tests that extracted bytecode is then handled properly. Would any internal refactoring of dis and addition of _test functions make it easier to make test more independent of each other?

4. I would rather read the multiple long lists like

expected_opinfo_outer = [
  Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval=3, argrepr='3', offset=0, starts_line=2, is_jump_target=False),
  Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, argrepr='4', offset=3, starts_line=None, is_jump_target=False),
  ...
]

condensed to

expected_opinfo_outer = [Instruction(opname, opcode, arg, argval, argrepr,
                                     offset, starts_line, is_jumps_target)
    for opname, opcode, arg, argval, argrepr, offset, starts_line, is_jumps_target in
(('LOAD_CONST', 100,    1,   3,     '3',      0,      2,           False),
 ('LOAD_CONST', 100,    2,   4,     '4',      3,      None,        False),
 ...
)]

[martindemello]

i'm interested in working on this (and refactoring dis to support better testability if needed). i like @terryjreedy 's ideas in general so i can start by implementing some of them.

[iritkatriel]

I think the most complex part of this issue is to make sure that the things that are covered by test_dis (and which do not belong there) are covered by other tests. Only then we can reduce test_dis to the right scope.

[terryjreedy]

I added checkboxes to Mark's todo list in the initial post. The first item is already fixed.

[martindemello]

there are still offsets in the Instruction(...) lists; I'll get rid of them in the next PR, so that this one can remain a pure refactoring.

[martindemello]

by this:

Contain big strings; write proper tests not just regex matches.

do you mean that we should always be matching the structured output of _get_instruction_bytes and not the formatted string output of _disassemble_bytes?