Fix truncated stream

[krassowski]

References

Fixes #450, enabling parsing of large responses from R server (e.g. completion on base) and any responses from javascript/typescript language servers

Code changes

Turns out BufferedIOBase.read() does buffering... And we need to read in chunks until we consume all needed chunks.

User-facing changes

Completion for R works better, completion for javascript works. May also help some Python packages with huge namespaces, e.g numpy. It will also improve perceived completion speed.

Backwards-incompatible changes

Chores

• linted
• tested - TODO a robot test for R base completion.
• documented
• changelog entry

[bollwyvl]

Awesome fix! I should probably figure out a python-side unit testing approach for more than just "does the server turn on"... While the acceptance tests are testing it, we'd probably get better insight (potentially benchmarks) from something with at least a _few_ layers not in the mix. Perhaps this is where the mock pygls server comes in, as we can more carefully tune our "worst case" examples, e.g. malformed messages, etc.

[krassowski]

It did not solve the issue on Windows which is slightly more complicated there.

On Windows we get OSError: [Errno 22] Invalid argument which seems to be coming from the non-blocking stream implementation (https://stackoverflow.com/questions/34504970/non-blocking-read-on-os-pipe-on-windows/34504971#34504971). Do we need the stream to be non-blocking? Should we rewrite it using tornado.iostream?

Also, the non-blocking behaviour has issues in Python: https://bugs.python.org/issue13322

Edit: removing the non-blocking mode fixes the issue on Windows.

[krassowski]

Also why do we use the BufferedIOBase in the first place? Could it be causing the weird performance issue for Window users? Just bouncing ideas I had when looking at the code....

[bollwyvl]

As long as we are running in the same event loop as the server, these really do need to be non-blocking. Now that we've dropped python 3.5 we might have some more options... I seem to recall that being a fairly major roadblock. The standard lib stuff may be enough... jupyter_server now has anyio as a dependency, which has some higher-level abstractions we also might be able use.