Asynchronous API, Networking and Asyncio Support

[lzpap]

Solves #224

Description

Motivation

So far, PyOTA has been relying on synchronous communication with nodes. This meant, that API requests could only be sent sequentially, one after the other. While this is simple and easy for developers to comprehend, it doesn't scale well and wastes a lot of precious time waiting for the results of network IO bound tasks.

Solution

With this PR, PyOTA is capable of doing asynchronous networking. Requests can be sent concurrently with the help of asyncio and httpx packages. When using the new AsyncIota API class, one can schedule API calls to execute concurrently. The API calls in the new class are actually python coroutines, but both their argument lists and return values are preserved from the previous version of PyOTA.

Regular, synchronous API calls are still supported with the "old" Iota API class. There should be no difference from a user perspective to issue a synchronous API call, however, the methods have been reworked to execute the aforementioned coroutines inside an event loop under the hood. This speeds up some extended API calls too.

Changes in source code

• Use AsyncClient from httpx for networking instead of requests. httpx supports Python 3.6+.

• Mimic the behavior of AsyncClient in MockAdapter, return an asyncio.Future response object for a request.

• Refactor RoutingWrapper to work with async.

• Create new API classes:

  • AsyncStrictIota and
  • AsyncIota.
    These API classes define the API calls as coroutines rather than methods. Hence, the API calls can be awaited by the application developer.

• Refactor StrictIota and Iota classes to build on top of the two new classes. They provide the synchronous method interface by executing the parents' coroutines in an event loop and returning their result.

• Refactor commands to work asynchronously. Modify BaseCommand.__call__() and BaseCommand._execute() to work as coroutines.

• All core commands are async.

• Update extended commands to work as async.

  • BroadcastAndStoreCommand is (nearly) twice as fast as before, which speeds up for example SendTrytesCommand and SendTransferCommand as well.
  • GetBundlesCommand is faster when called with multiple args.
  • GetNewAddressesCommand is faster as internal address checks are done concurrently.

• Remove Helpers class as IsPromotableCommand is properly implemented as an extended API call.

• Update iota.multisig module to work async, create new AsyncMultisigIota API class and rebase the original multisig api class on top of it, using the same logic as above.

• Remove Support for Pyhton 2 and Python 3.5!
  Async modules do not work well with 3.6>.

Changes in test code

• Make use of aiounittest module for testing async coroutines. @async_test decorator runs a test method inside an event loop.
• Update adapter and api tests.
• Update core API command tests for async.
• Update extended command tests for async.
• Update multisig tests for async.
• Remove PY2 and PY3.5 scenarios from test configuration.

Future work in another PR(s):

• Update documentation with new API classes and methods.
• Write examples to show how to use the new async classes.
• Clean up PY2 code snippets from codebase.
• Transform the codebase to PY3 format.

[todofixthis]

Looks good with minor questions/suggestions.

[todofixthis]

Suggested change

	super(StrictIota, self).add_neighbors(uris)
	super().add_neighbors(uris)

💭 Since we're no longer supporting Python 2, we can use py3-style super() calls.

[lzpap]

agreed 👍

[lzpap]

btw, @todofixthis could you explain me why super() is called in a base class?
For example here:

@add_metaclass(AdapterMeta)
class BaseAdapter(object):
    """
    Interface for IOTA API adapters.

    Adapters make it easy to customize the way an API instance
    communicates with a node.
    """
    supported_protocols = ()  # type: Tuple[Text]
    """
    Protocols that ``resolve_adapter`` can use to identify this adapter
    type.
    """

    def __init__(self):
        super(BaseAdapter, self).__init__()

[todofixthis]

🤔 Good question. I think the reason I did this is in case a derived class uses multiple inheritance, e.g., CustomAdapter(BaseAdapter, OtherBase): (or possibly the other way around; multiple inheritance is so confusing 😅) — without the super() call, it's possible that the constructor for one or more bases won't get called.

[lzpap]

Okay, thanks for the answer!

Let's say we switch to py3-style super() calls everywhere and don't need to explicitly tell type and object to super(). If I do it with BaseAdapter:

@add_metaclass(AdapterMeta)
class BaseAdapter:
    """
    Interface for IOTA API adapters.

    Adapters make it easy to customize the way an API instance
    communicates with a node.
    """
    supported_protocols = ()  # type: Tuple[Text]
    """
    Protocols that ``resolve_adapter`` can use to identify this adapter
    type.
    """

    def __init__(self):
        super().__init__()

I get a TypeError in the super() call:

Traceback (most recent call last):
  File "/pyota/test/commands/extended/utils_test.py", line 16, in setUp
    self.adapter = MockAdapter()
  File "/pyota/iota/adapter/__init__.py", line 547, in __init__
    super(MockAdapter, self).__init__()
  File "/pyota/iota/adapter/__init__.py", line 172, in __init__
    super().__init__()
TypeError: super(type, obj): obj must be an instance or subtype of type

, but the MRO seems okay:

>>> from iota import BaseAdapter
>>> BaseAdapter.__mro__
(<class 'iota.adapter.BaseAdapter'>, <class 'object'>)

What am I missing here? 🤔

[todofixthis]

Thanks for checking that out! Hmmm.. maybe it's not needed in Python 3 then? I don't recall the exact situation where I thought it was necessary; maybe we just remove it and see if anyone complains 😅