truncation of very long outputs

[anntzer]

Found while working on Calysto/matlab_kernel#5: wurlitzer truncates very long outputs. A minimal example is

Python 3.5.2 (default, Jun 28 2016, 08:46:01) 
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: %load_ext wurlitzer

In [2]: import ctypes; ctypes.CDLL(None).printf(b"%s", "\n".join(map(str, range(10000))).encode("
   ...: ascii"))

and the list of printed numbers stops around 280.

(It is even more annoying from the point of view of the MATLAB kernel as it simply hangs during the call to libc.fflush(c_stdout_p); I suspect the issues are related.)

[anntzer]

FWIW I worked around this by commenting out the calls fflush in the forwarder loop. I'm not sure they're necessary?

[minrk]

Thanks for the test!

They do seem to be necessary to ensure that you get all of the outputs at the right time. I'll see if there's a better way to accomplish that.

[minrk]

What OS are you on, BTW? I'm getting correct behavior when I run the example in a kernel on OS X. I do see the truncation when I run it in terminal IPython, though.

[minrk]

I just realized that it doesn't make any sense to use wurlitzer in terminal IPython, and I should make it raise if you try. Are you seeing this same truncation with this example in notebook/qtconsole?

[minrk]

I opened #2 which fixes the truncation in terminal IPython by disabling wurlitzer entirely in terminal IPython (there's nothing to capture when you are already in a terminal). I haven't been able to reproduce the truncation with a kernel, though.

[anntzer]

Hum, then the issue I was hitting with the MATLAB kernel was a separate one. Blockage also occurs with the qtconsole (with ipython/ipython#1392 appearing too). It does not seem to occur with the notebook though.

Edit: I am on Arch Linux.

[minrk]

There is nothing different about the kernel when running in the qtconsole vs the notebook, so I'm not sure how the kernel behavior could be different unless it happens to be running in a different environment (check sys.executable, sys.path). Perhaps it's just a coincidence and it only happens sometimes in both cases?

[anntzer]

Actually, I'm going to take back what I said: I also observe blockage with the notebook, just that the threshold seems different (higher). When blockage occurs, it occurs 100% of the times.

[minrk]

With the same ctypes command, or do you have another one that's more reliable for causing the block?

[anntzer]

With the MATLAB kernel (https://github.com/Calysto/matlab_kernel). Yes, I know, it's not easily reproducible if you don't have MATLAB available :-(