IDLE: Don't freeze Shell when user enters sys.stdout.shell.console

[HarrisonChudleigh]

BPO	34708
Nosy	@terryjreedy, @iritkatriel
Files	fail.py: After running this file, IDLE hangs or crashes, depending on version.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2018-09-17.02:53:01.502>
labels = ['3.10', 'expert-IDLE', '3.9', 'type-crash', '3.11']
title = 'Odd crashes/freezes when sys.stdout.shell.console is typed'
updated_at = <Date 2021-10-18.21:41:12.838>
user = 'https://bugs.python.org/HarrisonChudleigh'

bugs.python.org fields:

activity = <Date 2021-10-18.21:41:12.838>
actor = 'iritkatriel'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2018-09-17.02:53:01.502>
creator = 'Harrison Chudleigh'
dependencies = []
files = ['47808']
hgrepos = []
issue_num = 34708
keywords = []
message_count = 5.0
messages = ['325509', '325510', '325569', '404215', '404224']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'Harrison Chudleigh', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue34708'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-78889: Stop IDLE Shell freezes from sys.stdout.shell.xyz #121876
• [3.13] gh-78889: Stop IDLE Shell freezes from sys.stdout.shell.xyz (GH-121876) #121911
• [3.12] gh-78889: Stop IDLE Shell freezes from sys.stdout.shell.xyz (GH-121876) #121912

[HarrisonChudleigh]

If sys is imported and sys.stdout.shell.console is typed (this is what fail.py does TJR), IDLE does not return to the prompt. Ctrl-C has no effect and after a restart with Ctrl-F6, IDLE does not react after a command is typed in. This also occurs when other variables such as sys.stdout.shell.stdout are used, but not with non-existent variables such as sys.stdout.shell.not_a_variable. No such issue occurs using the command line; however, sys.stdout.shell is not defined in the command line. This behaviour exists in at least Python 3.6.1 and 3.7.0 on Mac OS X 10.9.

This might be related to a similar issue that occurs in at least Python 3.2 on Windows 2000. Entering sys.stdout.shell.console causes IDLE to crash immediately; a crash also occurs when you type sys.stdout.shell.console. and wait for IDLE to bring up the list of attributes.

I know that this bug shouldn't be encountered unless you're using the IDLE PyShell object for some reason, but it still seems weird that getting the value of a variable causes crashes and hangs, especially when I know that the variable exists from looking at idlelib.

This is marked as Python 3.4 to 3.7 because I tested with 3.7, 3.6 and 3.2 and I don't think that this would have been fixed and then broken again. I don't know about 3.8, though.

[HarrisonChudleigh]

Edit: I checked Python 3.4 (Windows XP SP3). It hangs in the same way as Python 3.6/3.7.

[terryjreedy]

3.4 and 3.5 only get security fixes and I doubt this qualifies.

I reproduced the described behavior with 3.7 on Win 10, so it is not Mac-specific.

Python sys.stdout: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
IDLE's sys.stdout: <idlelib.run.PseudoOutputFile object at 0x00000246821B1E10>
The PseudoFiles are unusual in being internal objects accessible from user code.

Comparing attributes with public names (not starting with single or double underscore) PseudoOutputFile lacks buffer, line_buffering, mode, reconfigure, and write_through. bpo-21995 is about fixing or documenting each omission for all 3 streams.

PseudoFiles add attributes 'shell' and 'tags'. Both should have private names that warn users that their behavior is undocumented and usage is at one's own risk. 'shell' should be double underscored. This minimal fix would be sufficient to close this issue. Some manual testing should be sufficient for such a change.

tags should be singular, as it is one of 'stdin', 'stdout', or 'stderr', and renamed stream. The tag is used in the write method to tell Shell whether to display with the configured stdout or stderr colors.

shell is an idlelib.rpc.RPCProxy object. As an object in itself, in the run process, it has 2 public attributes: oid ('console') and sockio (an idlelib.run.MyHandler object). But as a proxy representing the Shell console object in the IDLE process, it supposedly 'has' all the attributes of the latter. These are accessed with a custom __getattr__.

    def __getattr__(self, name):
        if self.__methods is None:
            self.__getmethods()
        if self.__methods.get(name):
            return MethodProxy(self.sockio, self.oid, name)
        if self.__attributes is None:
            self.__getattributes()
        if name in self.__attributes:
            value = self.sockio.remotecall(self.oid, '__getattribute__',
                                           (name,), {})
            return value
        else:
            raise AttributeError(name)

Note that self.__attributes is mangled to shell._RPCProxy__attributes when accessed externally. (Also, it is a set still implemented in this ancient code as a dict with int 1 values.)

More importantly, remote calls to the Shell console can only pass and return objects that can be pickled, and there is currently no provision for graceful failure. Hence sys.stdout.shell.width is (for me, currently) 80, but .console or .stdout are impossible to return.

I am extremely dubious about trying to fix this in the IDLE side of the link. Instead, the proxy should be initialized with sets of known usable attributes and methods. I think the underlying problem is trying to be unnecessarily and impossibly generic. Run only calls the .readline, .write, and .close methods of the console proxy. It only calls stack viewer setup on the interp proxy. A third proxy, for flist, is passed to the stack viewer (I have not yet checked what is needed for that). Restricting proxies to things that are needed and *should* work would make it more feasible to test them. But setting up automated tests would still require some work.

[iritkatriel]

Terry, I don't see any PseudoOutputFile in the current codebase. Is this out of date?

[terryjreedy]

Unfortunately, no. The standard stream replacements were renamed to StdOutputFile, etc, and slightly revised, but the buggy behavior remains. The worst is that shell restart does not completely restart the shell. It has to be closed instead.

I would like to fix this first by adding 'provision for graceful failure'. 'try' is now nearly free when there is no exception.

[terryjreedy]

• Handle bad (unpickleable) Shell attribute requests.
• Rename tags to _stream (to show that private) and shell to __shell (to mangle).
• Turn __attributes into set rather than dict.

[terryjreedy]

3.11.0b5 on Windows:

>>> sys.stdout.shell.console

... 
... 

After 1st Enter, cursor move to blank line and nothing happens. 2nd and 3rd Enters and '...' appears. This seems like a separate bug in secondary prompt mechanism. Otherwise, ^C has no effect, Restart appears to work in that the restart line and a prompt appear, but input does not work. Instead it goes into a secondary prompt loop. A fix is high priority.

[dlamblin]

If shell were renamed __shell would that mean breaking those few programs that try to find out if they're in an IDLE shell by doing something like:

from sys import stdout

MODE = None
try:
    MODE = stdout.shell
except AttributeError:
    pass

def mode_write(*args, sep=' ', end='\n', color='stdout'):
    "Prints with color if MODE is the IDLE Shell"
    if MODE:
        _ = MODE.write(sep.join(args) + end, color)
    else:
        print(*args, sep=sep, end=end)

if __name__ == '__main__':
    mode_write('This tends to be blue.')
    mode_write('This tends to be red.', color='COMMENT')
    mode_write('This tends to be green.', color='STRING')

[terryjreedy]

Yes. I have recommended import sys; "idlelib.run" in sys.modules for detecting IDLE but I have seen import sys; hasattr(sys.stdout, 'shell' or equivalent. While this is a private implementation detail and would normally not care, the fact that it is accessible without importing from idlelib suggests that I should either leave the name alone or leave the name with a dummy value.

[terryjreedy]

As near as I can tell, sys.stdxyz.shell is indeed the pyshell.shell instance of pyshell.PyShell, which inherits from outwin.OutputWindow. What is confusing is that shell has a console attribute, which is the pseudofile that prints to the input region of the Shell window, while shell becomes self.tkconsole in the pyshell.ModifiedInterpreter instance and is registered as 'console' in rpc. See MI.__init__, MI.start_subprocess, PyShell.__init__, run.MyHandler.handle, and bits in rpc.

I don't think the interp and flist proxies are accessible in the same way that the shell/console proxy is.

[terryjreedy]

Running IDLE from Command Prompt, sys.stdout.shell.console results in

Exception in Tkinter callback
Traceback (most recent call last):
  File "f:\dev\3x\Lib\tkinter\__init__.py", line 2068, in __call__
    return self.func(*args)
           ~~~~~~~~~^^^^^^^
  File "f:\dev\3x\Lib\tkinter\__init__.py", line 862, in callit
    func(*args)
    ~~~~^^^^^^^
  File "f:\dev\3x\Lib\idlelib\pyshell.py", line 582, in poll_subprocess
    response = clt.pollresponse(self.active_seq, wait=0.05)
  File "f:\dev\3x\Lib\idlelib\rpc.py", line 450, in pollresponse
    self.putmessage((seq, response))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\Lib\idlelib\rpc.py", line 335, in putmessage
    s = dumps(message)
  File "f:\dev\3x\Lib\idlelib\rpc.py", line 62, in dumps
    p.dump(obj)
    ~~~~~~^^^^^
TypeError: cannot pickle 'module' object

Should check currect doc whether TypeError instead of PicklingError is OK. p is a subclass of Pickler that should not change error types. The dumps call is wrapped as follows:

        try:
            s = dumps(message)
        except pickle.PicklingError:
            print("Cannot pickle:", repr(message), file=sys.__stderr__)
            raise

Will see what happens if TypeError is caught, or whether can convert to AttributeError or even string message.