Add examples to shell documentation

[mih]

#596 already has extensive documentation. However, there is a significant gap between this documentation and the ability to get something going.

I believe we need to add concrete examples on:

• run a command with stdin input (key advice is presently "hidden" in the docs of one argument)
• run a command with an expected failure
• perform a down/upload (fixed length)
• use the tool when it is impossible/impractical to go with a standard with block

And there need to be guidelines/examples on how to deal with different/unknown shell capabilities/platform detection.

Quite a few examples snippets already exist. It could be sufficient(?), but certainly useful to point them out explicitly in the top-level module docs.

[mih]

I looked at the docs again, trying to come up with a pattern or concept how things could be made more accessible. I thought it would be nice to have a "roll your own interpreter support" as an example to learn on the job.

Concretely, I think we could document what would be needed to support running a Python interpreter as a shell. Whether or not we would actually provide such functionality, it seems to be sufficiently different from present implementations that it would not "just work". A narrated example, with content like "why do we need this additional piece?", and "what base functionality can we customize?" would be rather insightful, I think.

[christian-monch]

I looked into it. Python is a challenge because it uses stderr as "signaling" channel. That means all prompts, e.g. >>> and ... are written to stderr as well as all error messages. Access to stderr output is limited in iterable_subprocess and consequently in iter_subproc and shell.

Having said that, if we only concentrate on stdout we can create an appropriate Response Generator relatively simple. The following code creates a response generator by subclassing VariableLengthResponseGenerator and adding appropriate zero_command and get_final_command-methods. In order to keep things simple, all exceptions are mapped onto the return code 1 and details have to be taken from the stderr-output of the results.

from datalad_next.shell import shell
from datalad_next.shell.response_generators import (
    VariableLengthResponseGenerator,
)


class PythonResponseGenerator(VariableLengthResponseGenerator):

    @property
    def zero_command(self) -> bytes:
        return b'True'

    def get_final_command(self, command: bytes) -> bytes:
        return f'''try:
    {command.decode()}
    print('{self.end_marker.decode()}')
    print(0)
except:
    print('{self.end_marker.decode()}')
    print(1)
    raise

'''.encode()


# a simple demo of `PythonResponseGenerator` 
with shell(['python', '-u', '-i'], zero_command_rg_class=PythonResponseGenerator) as py:
    for command in [
        '1 + 1',
        'a = 2 * 3',
        'a / 0',
        'a * 2',
    ]:
        r = py(command)
        print(command, r)

The resulting output is:

1 + 1 ExecutionResult(stdout=b'2\n', stderr=b'>>> ... ... ... ... ... ... ... ... ', returncode=0)
a = 2 * 3 ExecutionResult(stdout=b'', stderr=b'>>> ... ... ... ... ... ... ... ... >>> ', returncode=0)
a / 0 ExecutionResult(stdout=b'', stderr=b'... ... ... ... ... ... ... ... Traceback (most recent call last):\n  File "<stdin>", line 2, in <module>\nZeroDivisionError: division by zero', returncode=1)
a * 2 ExecutionResult(stdout=b'12\n', stderr=b'\n>>> ... ... ... ... ... ... ... ... >>> ', returncode=0)

This works in principle but lacks easily accessible error information. The error information could be added by rendering exceptions to stdout after the result indicator. This would require more code because we cannot reuse the send-method of VariableLengthResponseGenerator.

Is that the kind of example that you had in mind? We could describe how an extended response generator can reliably handle exceptions in another, more complex, example.