Enhance frame handing in warnings.warn() to skip by module rather that stacklevel

[doerwalter]

BPO	850482
Nosy	@doerwalter, @devdanzin, @jayvdb
Files	diff.txt diff2.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2003-11-27.23:12:08.000>
labels = ['type-feature', 'library']
title = 'Enhance frame handing in warnings.warn()'
updated_at = <Date 2015-12-10.08:16:31.826>
user = 'https://github.com/doerwalter'

bugs.python.org fields:

activity = <Date 2015-12-10.08:16:31.826>
actor = 'jayvdb'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-11-27.23:12:08.000>
creator = 'doerwalter'
dependencies = []
files = ['5709', '5710']
hgrepos = []
issue_num = 850482
keywords = ['patch']
message_count = 6.0
messages = ['44953', '44954', '44955', '44956', '82016', '256170']
nosy_count = 4.0
nosy_names = ['doerwalter', 'zseil', 'ajaksu2', 'jayvdb']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue850482'
versions = ['Python 3.2']

Linked PRs

• gh-39615: Add warnings.warn() skip_file_prefixes support #100840
• gh-39615: fix warning on return type mismatch #101407

[doerwalter]

This patch enhances warnings.warn() in the following
way: The stacklevel passed in may be negative. In that
case the call stack is searched for the innermost frame
whose module name differs in the first -stacklevel
components. This frame will be used in the report.

So when you have the following call stack:
m1.f()
m1.m11.f()
m1.m12.f()
m2.m21.f()
m2.m22.f()

and the innermost function() m2.m22.f() call
warnings.warn() with a stacklevel of -2 the frame
reported will be from m1.m12.f(), because it is the
first one from outside the m2 package.

[zseil]

I don't know how desireable this feature is, but
I think that a better solution would be to add
a new parameter to warnings.warn(), instead of
reusing the old one.

Also, your description is wrong; when warn() in
your example is called with stacklevel -2, the
reported frame is m2.m21.f(). To get your result,
you have to call warn() with stacklevel -1.

I think that a better solution would be to add
a skipmodules='moduleprefix' parameter to the
warn() function. warn() would then simply look
for the first frame whose module name doesn't
start with this prefix.

[doerwalter]

Here's an updated patch that implements your suggestion: skipmodules is a regular expression. Any module whose name matches the RE will be skipped when determining the stacklevel.
File Added: diff2.txt

[zseil]

Looks good to me. I like that skipmodules is a
regular expression, but I don't know how will it
affect the intended rewrite in C:

http://www.python.org/sf/1631171

It would be nice if the documentation mentioned
what is the interaction between stacklevel and
skipmodules parameters, at least something like:

"If you want to issue a warning outside your module,
and you don't know how many frames will have to be
skipped for that, pass some big integer for stacklevel"

If this patch is accepted, it should be put to use in
the _struct module:

>>> struct.Struct("<l").pack(4023029188) # note: module is sys
sys:1: DeprecationWarning: struct integer overflow masking is deprecated

[devdanzin]

Patch has tests, needs updating.

[jayvdb]

See similar http://bugs.python.org/issue25216 aka #69403

[gpshead]

Assigning to me to see if I get to this as I'm really wanting this feature while trying to add a warning deep within an an complicated package with multiple public API surfaces at different stacklevels.

[merwok]

Skipping by module seems good, but the implementation relies on file names — could it be module or package name?

[gpshead]

It seems like "by fully qualified module name" is what code authors might want as that's how we think... But in practice that might be hard to implement in a reliable manner. The __file__ module or package prefix match effectively lets you do that. Anticipated common use path prefix matching use cases: skip_file_prefixes=(__file__,) or skip_file_prefixes=(os.path.dirname(__file__),)

Investigating... The code objects in the frame appears to have a .co_qualname attribute, but it that always set?
Code objects are always defined within a file thus co_filename should be set for anything coming from a .py source file in a module or package (C extensions or manual code object creation could supply whatever "filename" they want, but those unusual things aren't cases I'd worry about matching and skipping), but code objects are not required to have names or qualified names (lambdas, comprehension expressions, nested functions, etc)...

It might be possible to get the base module name itself via frame.f_globals['__name__'] but even that sounds unreliable, and using just the __name__ as a match to skip is pretty non-specific.

I'll poke around, but I'm not sure if it is possible.

The other constraint I've been keeping in this code: It must be a fast check. warn() should not do a lot of computation.

[gpshead]

I don't believe module or package name is possible. Indeed frame.f_code.co_qualname doesn't give a module or package name. Frames and code objects don't have any reference back to a module or namespace. So I don't see any easy, fast, and reliable way to do that beyond the filename based match as my PR implements.

[gpshead]

I'm calling this fixed as it can skip by filename now which was a performant way to implement this.