Redesign Async client implementation with support to tornado and asyncio

[dhoomakethu]

Pymodbus async client now supports Tornado and asyncio along with twisted for async client. The user can choose the backend as per his needs. Sample examples as below.

from pymodbus.client.async import schedulers

# Import The clients

from pymodbus.client.async.tcp import AsyncModbusTCPClient as ModbusClient
from pymodbus.client.async.serial import AsyncModbusSerialClient as ModbusClient
from pymodbus.client.async.udp import AsyncModbusUDPClient as ModbusClient

# For tornado based async client use 
event_loop, future = ModbusClient(schedulers.IO_LOOP, port=5020)

# For twisted based async client use 
event_loop, future = ModbusClient(schedulers.REACTOR, port=5020)

# For asyncio based async client use 
event_loop, client = ModbusClient(schedulers.ASYNC_IO, port=5020)

# Here event_loop is a thread which would control the backend
# and future is a Future/deffered object which would be used to add call backs to run asynchronously.
# The Actual client could be accessed with future.result() with Tornado and future.result when using twisted

More detailed examples are available here

Note: asyncio implementation is still being ported from python3 branch and will be available only for python3

@moltob and @madsl I have used your asyncio code for tcp/udp , Please feel free to give a look. The unit tests are yet to be updated.

[cbergmiller]

Thanks for your work on this. It is much appreciated!
May i ask why loop.run_until_complete is used to create the Client in factory.tcp.async_io_factory?
What if the user wants to create ModbusClients on runtime when the asyncio loop is already running?

[dhoomakethu]

@cbergmiller the idea was to handle all the intricacies from with in the pymodbus library and provide the user a simple interface and that is the reason why the event loop is run beforehand, if you check the other factories we are following the same pattern. Your question is valid but I would consider it as an enhancement on top of the proposed change. I hope I have answered your question, if you think something could be improved please feel free to raise a PR/issue. Thanks

[pax0r]

Good work, I can't wait for this to be merged to mainstream.

[pax0r]

How using this helper is better than using normal asyncio interface? ;)

[dhoomakethu]

@pax0r didn't get your question, could you elaborate?

[pax0r]

I am not sure if it's a good idea to create helper function which consists only of single call to some standard lib element. I believe it may be better to just directly use asyncio.Future(). But it's a nit, you can ignore it ;)

[pax0r]

I am not sure if it's a good idea to create helper function which consists only of single call to some standard lib element. I believe it may be better to just directly use asyncio.Future(). But it's a nit, you can ignore it ;)

[pax0r]

This class does not have self.host (and host don't make sense for serial connection).

[dhoomakethu]

@pax0r thanks for taking the time to review , much appreciated. Lot of cleanup is still to be done with this branch. I am going to keep this PR for some more days for others to review as well and will take care of your comments.

[pax0r]

@dhoomakethu, I just started to work on project using asyncio and modbus communication and decided to give a try to your work. For now looks like asyncio part is working well, I'll come back if I will found any more problems.

[dhoomakethu]

Thanks , just curious, are you dealing with the serial part or the TCP? Please raise an issue if you come across any.

[pax0r]

For now Serial only for communication with Fatek PLCs, maybe I'll use TCP later on.

[cbergmiller]

I recently did some work on the Serial Asyncio Modbus Client. The API of serial_asyncio.create_serial_connection is different from the socket API. Also, the serial kwargs (baudrate etc.) were missing in AsyncioModbusSerialClient.__init__ Method. I didn't have time to test my changes because serial_asyncio does not yet work on windows. I will generate a pull request anyway for reference.

[dhoomakethu]

Will be merging this PR to dev branch by the EOW.

[makuser]

@dhoomakethu TBH I'm not quite sure how to implement this, since I'm using already using my own asyncio loop in my application.

Since It's using loop.run_until_complete in pymodbus/client/async/factory/tcp.py#L97, it would start a new loop on its own, which is not desirable and leads to a Runtime Exception. IMO it should give the option to attach to a loop that's already running.

Task exception was never retrieved
future: <Task finished coro=<BAS.async_start() done, defined at /srv/bas/lib/python3.5/site-packages/bas/core.py:70> exception=RuntimeError('This event loop is already running',)>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/srv/bas/lib/python3.5/site-packages/bas/core.py", line 72, in async_start
    await neuron.switch_to_async(self.loop, self.basConfig.alias_dict)
  File "/srv/bas/lib/python3.5/site-packages/bas/brain.py", line 141, in createClient
    loopi, clienti = ModbusClient(schedulers.ASYNC_IO, host=self.modbus_server, port=self.modbus_port, loop=self.loop)
  File "/srv/bas/lib/python3.5/site-packages/pymodbus/client/async/tcp.py", line 45, in __new__
    timeout=timeout, **kwargs)
  File "/srv/bas/lib/python3.5/site-packages/pymodbus/client/async/factory/tcp.py", line 97, in async_io_factory
    client = loop.run_until_complete(asyncio.gather(cor))[0]
  File "/usr/lib/python3.5/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 408, in run_forever
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

Or am I doing something wrong? I'm quite new to asyncio, so please tell me.

[cbergmiller]

@makuser you don't have to use the factory. You can import the async client directly from pymodbus.client.async.asyncio import ReconnectingAsyncioModbusTcpClient and use it with an already running loop. Example:

async def task(loop):
    client = AsyncioModbusTcpClient(loop=loop, host='127.0.0.1')
    await client.connect()
    rr = await client.protocol.read_holding_registers(1, 2, unit=1)
    print(rr)

[dhoomakethu]

@makuser, Can you provide a code snippet on how exactly you are trying to create the client ? Just in case you are not aware there are examples to create and use the various clients here, more specifically asyncio client example is here.

If you want to use existing loop that is already running, pass it as a kwarg loop.. This should work and I see you are already doing it. If it is failing then there must be something wrong going on , any code/logs would be helpful to debug the issue.

Update: The above scenario works when the asyncio loop is not yet started, in case you need to use the existing loop , please try what @cbergmiller has suggested.

[bhagn]

@dhoomakethu I think what @makuser is seeing is a genuine issue with the factory implementation. Ideally a new event loop should be created in the factory using asyncio.new_event_loop() and context set inside EventLoopThread.