codeop: 3.8.5 regression, warnings.simplefilter('error', SyntaxWarning) does not raise.

[Carreau]

BPO	41520
Nosy	@terryjreedy, @vstinner, @ambv, @tacaswell, @Carreau, @csabella, @miss-islington, @tirkarthi
PRs	bpo-41520: codeop no longer ignores SyntaxWarning #21838 [3.9] bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838) #21840 [3.8] bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838) #21841 bpo-41520: Fix second codeop regression #21848 [3.9] bpo-41520: Fix second codeop regression (GH-21848) #21860 [3.8] [3.9] bpo-41520: Fix second codeop regression (GH-21848) #21862

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-08-13.21:16:13.875>
created_at = <Date 2020-08-10.21:57:55.916>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10', 'release-blocker']
title = "codeop: 3.8.5 regression, warnings.simplefilter('error', SyntaxWarning) does not raise."
updated_at = <Date 2020-08-13.21:16:13.874>
user = 'https://github.com/Carreau'

bugs.python.org fields:

activity = <Date 2020-08-13.21:16:13.874>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2020-08-13.21:16:13.875>
closer = 'terry.reedy'
components = ['Interpreter Core']
creation = <Date 2020-08-10.21:57:55.916>
creator = 'mbussonn'
dependencies = []
files = []
hgrepos = []
issue_num = 41520
keywords = ['patch']
message_count = 19.0
messages = ['375152', '375153', '375154', '375159', '375223', '375230', '375231', '375232', '375236', '375252', '375253', '375273', '375301', '375312', '375322', '375324', '375328', '375329', '375348']
nosy_count = 8.0
nosy_names = ['terry.reedy', 'vstinner', 'lukasz.langa', 'tcaswell', 'mbussonn', 'cheryl.sabella', 'miss-islington', 'xtreak']
pr_nums = ['21838', '21840', '21841', '21848', '21860', '21862']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41520'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[Carreau]

assuming

    $ cat foo.py
    import warnings
    from codeop import compile_command

    warnings.simplefilter('error', SyntaxWarning)
    res = compile_command('1 is 1\n', symbol='exec')
    print('Res', res)

On 3.8.0...3.8.4 this correctly raises a SyntaxError:

 python  foo.py
Traceback (most recent call last):
  File "foo.py", line 5, in <module>
    res = compile_command('1 is 1\n', symbol='exec')
  File "/Users/bussonniermatthias/miniconda3/envs/38/lib/python3.8/codeop.py", line 122, in compile_command
    return _maybe_compile(_compile, source, filename, symbol)
  File "/Users/bussonniermatthias/miniconda3/envs/38/lib/python3.8/codeop.py", line 99, in _maybe_compile
    raise err1
  File "/Users/bussonniermatthias/miniconda3/envs/38/lib/python3.8/codeop.py", line 87, in _maybe_compile
    code1 = compiler(source + "\n", filename, symbol)
  File "/Users/bussonniermatthias/miniconda3/envs/38/lib/python3.8/codeop.py", line 102, in _compile
    return compile(source, filename, symbol, PyCF_DONT_IMPLY_DEDENT)
  File "<input>", line 1
SyntaxError: "is" with a literal. Did you mean "=="?

But will silently return None on 3.8.5

$ python  foo.py
Res None

[Carreau]

seem to affect 3.8.4 as well.

[Carreau]

Potentially due to

https://bugs.python.org/issue40807
#20486

[tacaswell]

bisecting agrees with Matthias:

# first bad commit: [c067183] bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486)

[vstinner]

I proposed a fix: PR 21838. Would you mind to review it?

[vstinner]

New changeset 369a1cb by Victor Stinner in branch 'master':
bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838)
369a1cb

[miss-islington]

New changeset afff51f by Miss Islington (bot) in branch '3.8':
bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838)
afff51f

[miss-islington]

New changeset 90eff4e by Miss Islington (bot) in branch '3.9':
bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838)
90eff4e

[vstinner]

Thanks Matthias Bussonnier for the fix. The regression should now be fixed. Also, I converted your reproducer into a regression test.

[Carreau]

Thanks Matthias Bussonnier for the fix.

Thank *YOU* for the fix, and the bug report is initially from tcaswell.

At least on 3.8 branch this is fixed for me.

Just for completeness, this was discovered as in IPython we try to guess whether "enter" is "insert new line" or "execute", and 1 is 1<enter> would keep adding new lines.

Much love for the fast turnaround; looking fwd to 3.9

[vstinner]

Thanks tcaswell in this case ;-)

[terryjreedy]

This 'fix' introduces another regression by undoing the previous fix in bpo-40807 of emitting DeprecationWarning (for '\e', for instance) just once instead of thrice.

I suspect that the example of Matthias would fail in a debug build (with DeprecationWarnings on) for DeprecationWarning also.

PR-21848 removes the limitation of the first fix to SyntaxWarning and includes DeprecationWarning in the test for single emission.

[tirkarthi]

The commit 369a1cb introduces a change in environment. I haven't checked the other PR though to see if this is fixed.

./python.exe -m test --fail-env-changed test_codeop -m test_warning
0:00:00 load avg: 2.30 Run tests sequentially
0:00:00 load avg: 2.30 [1/1] test_codeop
Warning -- warnings.filters was modified by test_codeop
Before: (4331716096, [('default', None, <class 'DeprecationWarning'>, '__main__', 0), ('ignore', None, <class 'DeprecationWarning'>, None, 0), ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0), ('ignore', None, <class 'ImportWarning'>, None, 0), ('ignore', None, <class 'ResourceWarning'>, None, 0)], [('default', None, <class 'DeprecationWarning'>, '__main__', 0), ('ignore', None, <class 'DeprecationWarning'>, None, 0), ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0), ('ignore', None, <class 'ImportWarning'>, None, 0), ('ignore', None, <class 'ResourceWarning'>, None, 0)])
After: (4331716096, [('default', None, <class 'DeprecationWarning'>, '__main__', 0), ('ignore', None, <class 'DeprecationWarning'>, None, 0), ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0), ('ignore', None, <class 'ImportWarning'>, None, 0), ('ignore', None, <class 'ResourceWarning'>, None, 0)], [('error', None, <class 'SyntaxWarning'>, None, 0), ('default', None, <class 'DeprecationWarning'>, '__main__', 0), ('ignore', None, <class 'DeprecationWarning'>, None, 0), ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0), ('ignore', None, <class 'ImportWarning'>, None, 0), ('ignore', None, <class 'ResourceWarning'>, None, 0)])
test_codeop failed (env changed)

== Tests result: ENV CHANGED ==

1 test altered the execution environment:
test_codeop

Total duration: 38 ms
Tests result: ENV CHANGED

On an unrelated note the test passes with verbose mode though fail-env-changed flag is passed.

./python.exe -m test --fail-env-changed test_codeop -m test_warning -vvv
== CPython 3.10.0a0 (heads/master:c51db0ea40d, Aug 13 2020, 15:47:27) [Clang 10.0.1 (clang-1001.0.46.4)]
== macOS-10.14.6-x86_64-i386-64bit little-endian
== cwd: /Users/kasingar/stuff/python/cpython/build/test_python_29945æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.53 Run tests sequentially
0:00:00 load avg: 1.53 [1/1] test_codeop
test_warning (test.test_codeop.CodeopTests) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.001s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 58 ms
Tests result: SUCCESS

[vstinner]

The commit 369a1cb introduces a change in environment. (...)

Right. It's fixed by PR 21848.

[terryjreedy]

New changeset c818b15 by Terry Jan Reedy in branch 'master':
bpo-41520: Fix second codeop regression (GH-21848)
c818b15

[vstinner]

New changeset c818b15 by Terry Jan Reedy in branch 'master':
bpo-41520: Fix second codeop regression (GH-21848)
c818b15

I confirm that this change fixed the "test altered the execution environment" issue.

[terryjreedy]

New changeset f24430f by Terry Jan Reedy in branch '3.9':
[3.9] bpo-41520: Fix second codeop regression (GH-21848)
f24430f

[miss-islington]

New changeset a3416c1 by Miss Islington (bot) in branch '3.8':
[3.9] bpo-41520: Fix second codeop regression (GH-21848)
a3416c1

[terryjreedy]

Lukasz, please cherry-pick the 3.9 commits for this issue
90eff4e
f24430f
into 3.9.0rc2 before release.
They fix a regression introduced by a bug fix last June.