This commit adds the parameter `bufsize` to
`iterable_subprocesses` and to `iter_subproc`.
Its value is passed into `subprocess.Popen`.

This change is necessary to implement
interactions with interactive subprocesses,
for example a remote shell. These
processes usually require a buffer size
of zero, because each "request", i.e. an
input to an interactive process, results
in a "response" of unknown size and
structure. In an interactive communication,
each repsonse has to be interpreted before
the next request can be sent. Therefore
every byte that an interactive process emits
has to be sent to the client, which
processes it and assembles a response.

In addition this commit makes the previously
internal class `OutputFrom` (defined in
`datalad_next.iterable_subprocess.iterable_subprocess`)
available as
`datalad_next.iterable_subprocess.OutputFrom`.

This commit is in preparation for an
interactive shell-class that will be
used, for example, in RIA stores.

This commit improves the performance
of `align_pattern` by using a compiled
regular expression to match pattern
prefixes at the end of chunks. This
increases the performance of
`align_pattern`, roughly by
the factor 8.

The `shell` context manager
opens a connection to a shell. Typically
the shell is executed remotely via ssh,
but it might also be a local shell, e.g. `bash`.
It allows to send commands to the shell,
read the output of `stdout` and determines
the return code of the command.
When the context is left, the shell is
closed, by closing its `stdin`.

The `shell` context manager is extensible
through the concept of "Response Generators".
New response generator-classes can be used
to support new operating system, differing
command line syntax, or different behavior
of commands on different operating system.
Currently it supports `bash` and `PowerShell`.

The commit also includes a number of
"convenience" functions for Linux and OSX
that can be executed on a `shell`-context.

 - download files
 - upload files
 - delete files

This commit incoroporates
fixes and suggestions from:

Adina Wagner <[email protected]>
Michael Hanke <[email protected]>

Many thanks for that.

This commit adds a safe_read wrapper to prevent
a `ValueError`-exception if the thread
`iterable_subprocess.<locals>.input_to` tries
to read from a closed file.

A situation that triggers this error is
described here:

<datalad#596 (comment)>

This commit modifies the code to always raise
a `CommandError`, if a command that is
executed in a shell fails. This is in line
with the behavior of the other runner-tools.

This commit addressed a reviewer comment that
auggested to also accept commands given as
`str`. This leads to an extension of the
`ShellCommandExecutor.__call__`-interface
by the new parameter `encoding` that
defines the encoding that should be used
when converting `str`-commands into
`bytes`-commands.

`ShellCommandResponseGenerator.get_command_list`
is renamed to
`ShellCommandResponseGenerator.get_final_command`
and its return value is now `bytes` instead of a
list of bytes.

Some duplicated code in response generators has
been moved into base classes.

`ShellCommandResponseGenerator.get_number_and_newline`
was made private since it is only used by
subclasses of `ShellCommandResponseGenerator`.

This commit fixed an error in `str`-command
handling and adds testing for `str`-commands.

This commit adds a queue to upload that is used
by an iterator to signal the completion of the
read-operation on the local file. This is a better
way to prevent the "read() on closed file"-error.

A docstring-error is fixed as well.

This commit uses `ls -dln` instead of
`stat` to determine the size of a
file on the connected shell. This is
supported by more Unix-flavors, e.g.
by Linux and OSX.

TODO: update docstrings

1. make shell.__call__ consume everything
2. add shell.start
3. don't raise CommandError by default
4. add check to enable CommandError-raising
5. small improvements
6. extend tests

Previously, this has been a series of SSH calls, each running in its own
shell. Now a persistent shell is used to interleave remote and local
checks.

This commit adapts the changes made by
mih in his forked branch to the changes
that were made in this branch in the
meantime.

Co-authored-by: Michael Hanke <[email protected]>