asyncio server implementation #400
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When writing to broadcast address (unit_id=0) there should be no response according to the Modbus spec. This fix changes expected_response_length to 0 when writing to unit_id=0. This will break any existing code that is improperly using unit_id 0 for a slave address.
Fix pymodbus-dev#366 Update failures in sql context Update Changelog Fix major minor version in example codes
…ing packet. 2. Fix asyncio examples. 3. Minor update in factory.py, now server logs prints received request instead of only function cod
Adds broadcast_enable parameter to client and server, default value is False. When true it will treat unit_id 0 as broadcast and execute requests on all server slave contexts and not send a response and on the client side will send the request and not try to receive a response.
If the CRC recieved is not correct in my case my slave got caught in a deadlock, not taking any new requests. This addition fixed that.
Alternate solution for pymodbus-dev#356 and pymodbus-dev#360. Changes the RTU to make the transaction ID as the unit ID instead of an ever incrementing number. Previously this transaction ID was always 0 on the receiving end but was the unique transaction ID on sending. As such the FIFO buffer made the most sense. By tying it to the unit ID, we can recover from failure modes such as: - - Asyncio task cancellations (eg. timeouts) pymodbus-dev#360 - Skipped responses from slaves. (hangs on master pymodbus-dev#360) - CRC Errors pymodbus-dev#356 - Busy response
… version to 2.2.0rc4
|
@memetb Thanks for the PR, Could you please fix the failing tests ? |
|
Sure. At least one failures is that I have 0% code coverage. Like I said, I will tackle this time permitting this week but I would appreciate if you point me to existing tests that i could reuse. Thanks. |
|
You can take a look in to existing async_server tests |
|
@dhoomakethu sorry this has taken so long to advance. Do you have a problem with adding an asynctest dependency to your tests? This is the package I use for making asyncio native unittests. |
|
@dhoomakethu I have added the unittests for the asycnio server implementation. Known issues:
Minor point: commit 50c5117 was a minor detail to cleanup code base, as the project was failing to load on an install without serial. |
|
@memetb Thanks for the updates, I will take a look and will create a separate issue for serial server. |
…t's execute method
|
Does this work? |
|
Pleased to see this work progressing! I think you need to add asynctest to requirements-tests.txt to fix the CI errors. |
…ssertion from unsupported targets, use lower asynctest version
|
I've added the asynctest to the requirements file and also skipped tests for outright unsupported targets (<3.5), however, on targets that don't support it (e.g. python2), the loading of the unittest fails because of the Furthermore: on higher targets, there is also a version mismatch between earlier versions of asyncio and the current 3.7 which I have coded and tested against. In the travis config file, the 3.7 target is specifically commented out which I guess is because the overall library isn't ready for it. Let me know what you guys think. I'm personally not enormously upset by lack of backward compatibility since asyncio is a new feature and pymodbus has worked without it before this. However, if the project requirement for this feature be that it's as far backward compatible as possible, I'm going to have to go back to the drawing board and this may take a few more weeks. |
…mers' already set log level
…loaded in py2 even if the test is to be skipped
|
Ok, I think a more extensive refactoring is required. I just pushed a commit removing 3.7 language keywords from the unittests and now there's issues with f-strings. Still open to suggestions. |
|
@memetb I am yet to go through the code, but at present we skip the unittests for python<3.4 with this fixture |
|
Thanks @dhoomakethu. Commit 985d0cf added just that fixture. As an aside, I've been looking at some other projects (e.g. influxdb python client) and I might get some ideas. Also, pending my surplus time from work, I've come to think it's probably better for me to update the implementation itself so that it works on earlier asyncio versions given that this is a library and not an end-user application. Comments are welcome, as always. |
|
Any updates on this PR? |
|
@memetb I am waiting for the tests to be fixed. |
|
@dhoomakethu I'll be going on leave for a short bit so won't be working on anything before September. However, please check my comments above: code coverage is upper 90's and there are unit tests for each feature. What is failing to build is that the project itself does not have python 3.7 support and this code is not meant for (or even useful for) any language level below that. |
|
@memetb thanks for the clarification, I will merge this and fix the tests later. |
dhoomakethu
approved these changes
Jul 29, 2019
|
@memetb did you close this by Mistake? |
|
@dhoomakethu: yes, sorry about that. |
dhoomakethu
pushed a commit
that referenced
this pull request
Oct 17, 2019
* client/sync.py: Fix missing serial module dependency
The serial.connect failed in PR riptideio#400 with "NameError: name
'serial' is not defined" [1]:
self = <ModbusSerialClient at 0x7fcda4009b00 socket=None, method=ascii, timeout=3>
def connect(self):
""" Connect to the modbus serial server
:returns: True if connection succeeded, False otherwise
"""
if self.socket:
return True
try:
> self.socket = serial.Serial(port=self.port,
timeout=self.timeout,
bytesize=self.bytesize,
stopbits=self.stopbits,
baudrate=self.baudrate,
parity=self.parity)
E NameError: name 'serial' is not defined
pymodbus/client/sync.py:476: NameError
This patch moves the serial import back to the head.
[1] https://travis-ci.org/riptideio/pymodbus/jobs/566009109
Fixes: commit e6da559 asyncio server implementation (#400)
* server/asyncio.py: Create server with appropriate args and environment
If Python is older than 3.7, the create_server will fail like PR
riptideio#400 with "unexpected keyword argument 'start_serving'" [1]
which is new in Python version 3.7:
self.server_factory = self.loop.create_server(lambda :self.handler(self),
*self.address,
reuse_address=allow_reuse_address,
reuse_port=allow_reuse_port,
backlog=backlog,
> start_serving=not defer_start)
E TypeError: create_server() got an unexpected keyword argument 'start_serving'
pymodbus/server/asyncio.py:400: TypeError
This patch creates server according to Python environment.
[1] https://travis-ci.org/starnight/pymodbus/jobs/584178484
Fixes: commit e6da559 asyncio server implementation (#400)
* Create asyncio task with appropriate method and environment
If Python is older than 3.7, the asyncio.create_task will fail like PR
riptideio#400 with "AttributeError: module 'asyncio' has no attribute
'create_task'" [1] which is new in Python version 3.7 [2]:
@asyncio.coroutine
def testTcpServerCloseActiveConnection(self):
''' Test server_close() while there are active TCP connections'''
data = b"\x01\x00\x00\x00\x00\x06\x01\x01\x00\x00\x00\x01"
server = yield from StartTcpServer(context=self.context,address=("127.0.0.1", 0),loop=self.loop)
> server_task = asyncio.create_task(server.serve_forever())
E AttributeError: module 'asyncio' has no attribute 'create_task'
test/test_server_asyncio.py:205: AttributeError
This patch creates task according to Python environment.
[1] https://travis-ci.org/starnight/pymodbus/jobs/584193587
[2] https://docs.python.org/3/library/asyncio-task.html#creating-tasks
Fixes: commit e6da559 asyncio server implementation (#400)
* server/asyncio.py: Fix format string for older Python
If Python is older than 3.6, f-Strings will fail like PR riptideio#400
with "SyntaxError: invalid syntax" [1] which is new in Python version
3.6 with PEP 498 -- Literal String Interpolation [2]:
test/test_server_asyncio.py:14: in <module>
from pymodbus.server.asyncio import StartTcpServer, StartUdpServer, StartSerialServer, StopServer, ModbusServerFactory
E File "/home/travis/build/starnight/pymodbus/pymodbus/server/asyncio.py", line 424
E _logger.warning(f"aborting active session {k}")
E ^
E SyntaxError: invalid syntax
This patch fixes the format string with traditional format string
syntax.
[1] https://travis-ci.org/starnight/pymodbus/jobs/584427976
[2] https://www.python.org/dev/peps/pep-0498/
Fixes: commit e6da559 asyncio server implementation (#400)
* test: Make assert_called_once() test only with Python 3.6+
If Python is older than 3.6, unittest.mock.assert_called_once() will
fail like PR riptideio#400 with "AttributeError: assert_called_once" [1]
which is new in Python version 3.6 [2]:
> self.loop.create_server.assert_called_once()
test/test_server_asyncio.py:76:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <CoroutineMock name='mock.create_server' id='139638313234784'>
name = 'assert_called_once'
def __getattr__(self, name):
if name in {'_mock_methods', '_mock_unsafe'}:
raise AttributeError(name)
elif self._mock_methods is not None:
if name not in self._mock_methods or name in _all_magics:
raise AttributeError("Mock object has no attribute %r" % name)
elif _is_magic(name):
raise AttributeError(name)
if not self._mock_unsafe:
if name.startswith(('assert', 'assret')):
> raise AttributeError(name)
E AttributeError: assert_called_once
/opt/python/3.5.6/lib/python3.5/unittest/mock.py:585: AttributeError
This patch skips the tests if they are not in Python 3.6+.
[1] https://travis-ci.org/starnight/pymodbus/jobs/584431003
[2] https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.assert_called_once
Fixes: commit e6da559 asyncio server implementation (#400)
* test: Make serve_forever() test only with Python 3.7+
If Python is older than 3.7, asyncio.base_events.Server.serve_forever
will fail like PR riptideio#400 with "AttributeError: <class
'asyncio.base_events.Server'> does not have the attribute
'serve_forever'" [1] which is new in Python version 3.7 [2]:
@asyncio.coroutine
def testTcpServerServeNoDefer(self):
''' Test StartTcpServer without deferred start (immediate execution of server) '''
> with patch('asyncio.base_events.Server.serve_forever', new_callable=asynctest.CoroutineMock) as serve:
test/test_server_asyncio.py:81:
...
if not self.create and original is DEFAULT:
raise AttributeError(
> "%s does not have the attribute %r" % (target, name)
)
E AttributeError: <class 'asyncio.base_events.Server'> does not have the attribute 'serve_forever'
This patch skips the tests if they are not in Python 3.7+.
[1] https://travis-ci.org/starnight/pymodbus/jobs/584212511
[2] https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Server.serve_forever
Fixes: commit e6da559 asyncio server implementation (#400)
dhoomakethu
added a commit
that referenced
this pull request
Oct 29, 2019
* Rebase to dev3.7 * Adding 3.7 to travis configuration * Updated documentation to resolve warnings introduced with the longer names Updated requirements-docs.txt to include missing modules * Fixed reference to deprecated asynchronous * Adding gmp disable to fix pypy build issues * Adding gmp disable to fix pypy build issues * Removing travis python 3.7 configuration Commenting out python3.7 from Travis while waiting for support. You can run teh 3.7 tests with tox without issues * Adding asserts for Payload Endianness * Fixing example of Payload. Same Endianness for builder and decoder. * Fix Sql db slave context validate and get methods - #139 * #353 - debugging, Add debug logs to check size of avaialble data in read buffer * #353 Provide an option to disable inter char timeouts * #353 Bump version, update changelog * check self.socket (#354) * check self.socket self.socket might be None at this point * Update pymodbus/client/sync.py Co-Authored-By: mpf82 <[email protected]> * Fix typo (#378) * Pymodbus 2.2.0 (#375) * #357 Support registration of custom requests * #368 Fixes write to broadcast address When writing to broadcast address (unit_id=0) there should be no response according to the Modbus spec. This fix changes expected_response_length to 0 when writing to unit_id=0. This will break any existing code that is improperly using unit_id 0 for a slave address. * Bump version to 2.2.0 Fix #366 Update failures in sql context Update Changelog Fix major minor version in example codes * Fix #371 pymodbus repl on python3 * 1. Fix tornado async serial client `TypeError` while processing incoming packet. 2. Fix asyncio examples. 3. Minor update in factory.py, now server logs prints received request instead of only function cod * [fix v3] poprawa sprawdzania timeout * Release candidate for pymodbus 2.2.0 * Fix #377 when invalid port is supplied and minor updates in logging * #368 adds broadcast support for sync client and server Adds broadcast_enable parameter to client and server, default value is False. When true it will treat unit_id 0 as broadcast and execute requests on all server slave contexts and not send a response and on the client side will send the request and not try to receive a response. * #368 Fixes minor bug in broadcast support code * Fixed erronous CRC handling If the CRC recieved is not correct in my case my slave got caught in a deadlock, not taking any new requests. This addition fixed that. * Update Changelog * Fix test coverage * Fix #387 Transactions failing on 2.2.0rc2. * Task Cancellation and CRC Errors Alternate solution for #356 and #360. Changes the RTU to make the transaction ID as the unit ID instead of an ever incrementing number. Previously this transaction ID was always 0 on the receiving end but was the unique transaction ID on sending. As such the FIFO buffer made the most sense. By tying it to the unit ID, we can recover from failure modes such as: - - Asyncio task cancellations (eg. timeouts) #360 - Skipped responses from slaves. (hangs on master #360) - CRC Errors #356 - Busy response * Cherry pick commit from PR #367 , Update changelog , bump version to 2.2.0rc4 * #389 Support passing all serial port parameters to asynchronous server * Fix BinaryPayloadDecoder and Builder wrt to coils * Misc updates, bump version to 2.2.0 * ReportSlaveIdResponse now tries to get slave id based on server identity for pymodbus servers * Update missing bcrypt requirement for testing * Fix docs (#407) * Fix document generation * Formatting updates in Changelog * Remove pycrypto dep (#411) It has not been needed by Twisted for a long time, and has been unmaintained for a long time. * Fix --upgrade option in install dependencies (#413) * Fix document generation * Formatting updates in Changelog * Fix --upgrade option in install dependencies * Padding for odd sized responses (#425) If the response is odd size the buffer needs to be padded with an additional byte. * README update: REPL stands for Read Evaluate **Print** Loop (#426) * Drop python 3.4 support (#440) Python 3.4 is EoL and has an easy upgrade path to 3.5+. Support was dropped in Twisted 19.7.0, which is causing Travis to fail pymodbus tests for 3.4. * Re-enable travis python 3.7 builds (#441) * Update __init__.py (#436) * Use SPDX identifier to specify the exact license type (#427) * asyncio server implementation (#400) * #357 Support registration of custom requests * #368 Fixes write to broadcast address When writing to broadcast address (unit_id=0) there should be no response according to the Modbus spec. This fix changes expected_response_length to 0 when writing to unit_id=0. This will break any existing code that is improperly using unit_id 0 for a slave address. * Bump version to 2.2.0 Fix #366 Update failures in sql context Update Changelog Fix major minor version in example codes * Fix #371 pymodbus repl on python3 * 1. Fix tornado async serial client `TypeError` while processing incoming packet. 2. Fix asyncio examples. 3. Minor update in factory.py, now server logs prints received request instead of only function cod * [fix v3] poprawa sprawdzania timeout * Release candidate for pymodbus 2.2.0 * Fix #377 when invalid port is supplied and minor updates in logging * #368 adds broadcast support for sync client and server Adds broadcast_enable parameter to client and server, default value is False. When true it will treat unit_id 0 as broadcast and execute requests on all server slave contexts and not send a response and on the client side will send the request and not try to receive a response. * #368 Fixes minor bug in broadcast support code * Fixed erronous CRC handling If the CRC recieved is not correct in my case my slave got caught in a deadlock, not taking any new requests. This addition fixed that. * Update Changelog * Fix test coverage * Fix #387 Transactions failing on 2.2.0rc2. * Task Cancellation and CRC Errors Alternate solution for #356 and #360. Changes the RTU to make the transaction ID as the unit ID instead of an ever incrementing number. Previously this transaction ID was always 0 on the receiving end but was the unique transaction ID on sending. As such the FIFO buffer made the most sense. By tying it to the unit ID, we can recover from failure modes such as: - - Asyncio task cancellations (eg. timeouts) #360 - Skipped responses from slaves. (hangs on master #360) - CRC Errors #356 - Busy response * Cherry pick commit from PR #367 , Update changelog , bump version to 2.2.0rc4 * native asyncio implementation of ModbusTcpServer and ModbusUdpServer * preliminary asyncio server examples * move serial module dependency into class instantiation * unittests for asyncio based server implementation * induce exception in execute method by mock patching the request object's execute method * move serial module dependency into class instantiation * added asynctest depency to requirements-tests.txt * add unittest skip condition for unsupported targets, remove failing assertion from unsupported targets, use lower asynctest version * remove logger setLevel call since doing so may override library consumers' already set log level * remove async def/await keywords from unittest so that the ast can be loaded in py2 even if the test is to be skipped * Add option to repl allowing Modbus RTU framing on a TCP socket (#447) * repl: Allow Modbus RTU framing on a TCP socket * repl: Update README for framing option * Fix asynci server test failures on python3.6 and below * Bump version to 2.2.0rc1, update six requirements and Changelog * Support multiple Python versions to fix test error from PR #400 (#444) * client/sync.py: Fix missing serial module dependency The serial.connect failed in PR riptideio#400 with "NameError: name 'serial' is not defined" [1]: self = <ModbusSerialClient at 0x7fcda4009b00 socket=None, method=ascii, timeout=3> def connect(self): """ Connect to the modbus serial server :returns: True if connection succeeded, False otherwise """ if self.socket: return True try: > self.socket = serial.Serial(port=self.port, timeout=self.timeout, bytesize=self.bytesize, stopbits=self.stopbits, baudrate=self.baudrate, parity=self.parity) E NameError: name 'serial' is not defined pymodbus/client/sync.py:476: NameError This patch moves the serial import back to the head. [1] https://travis-ci.org/riptideio/pymodbus/jobs/566009109 Fixes: commit e6da559 asyncio server implementation (#400) * server/asyncio.py: Create server with appropriate args and environment If Python is older than 3.7, the create_server will fail like PR riptideio#400 with "unexpected keyword argument 'start_serving'" [1] which is new in Python version 3.7: self.server_factory = self.loop.create_server(lambda :self.handler(self), *self.address, reuse_address=allow_reuse_address, reuse_port=allow_reuse_port, backlog=backlog, > start_serving=not defer_start) E TypeError: create_server() got an unexpected keyword argument 'start_serving' pymodbus/server/asyncio.py:400: TypeError This patch creates server according to Python environment. [1] https://travis-ci.org/starnight/pymodbus/jobs/584178484 Fixes: commit e6da559 asyncio server implementation (#400) * Create asyncio task with appropriate method and environment If Python is older than 3.7, the asyncio.create_task will fail like PR riptideio#400 with "AttributeError: module 'asyncio' has no attribute 'create_task'" [1] which is new in Python version 3.7 [2]: @asyncio.coroutine def testTcpServerCloseActiveConnection(self): ''' Test server_close() while there are active TCP connections''' data = b"\x01\x00\x00\x00\x00\x06\x01\x01\x00\x00\x00\x01" server = yield from StartTcpServer(context=self.context,address=("127.0.0.1", 0),loop=self.loop) > server_task = asyncio.create_task(server.serve_forever()) E AttributeError: module 'asyncio' has no attribute 'create_task' test/test_server_asyncio.py:205: AttributeError This patch creates task according to Python environment. [1] https://travis-ci.org/starnight/pymodbus/jobs/584193587 [2] https://docs.python.org/3/library/asyncio-task.html#creating-tasks Fixes: commit e6da559 asyncio server implementation (#400) * server/asyncio.py: Fix format string for older Python If Python is older than 3.6, f-Strings will fail like PR riptideio#400 with "SyntaxError: invalid syntax" [1] which is new in Python version 3.6 with PEP 498 -- Literal String Interpolation [2]: test/test_server_asyncio.py:14: in <module> from pymodbus.server.asyncio import StartTcpServer, StartUdpServer, StartSerialServer, StopServer, ModbusServerFactory E File "/home/travis/build/starnight/pymodbus/pymodbus/server/asyncio.py", line 424 E _logger.warning(f"aborting active session {k}") E ^ E SyntaxError: invalid syntax This patch fixes the format string with traditional format string syntax. [1] https://travis-ci.org/starnight/pymodbus/jobs/584427976 [2] https://www.python.org/dev/peps/pep-0498/ Fixes: commit e6da559 asyncio server implementation (#400) * test: Make assert_called_once() test only with Python 3.6+ If Python is older than 3.6, unittest.mock.assert_called_once() will fail like PR riptideio#400 with "AttributeError: assert_called_once" [1] which is new in Python version 3.6 [2]: > self.loop.create_server.assert_called_once() test/test_server_asyncio.py:76: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <CoroutineMock name='mock.create_server' id='139638313234784'> name = 'assert_called_once' def __getattr__(self, name): if name in {'_mock_methods', '_mock_unsafe'}: raise AttributeError(name) elif self._mock_methods is not None: if name not in self._mock_methods or name in _all_magics: raise AttributeError("Mock object has no attribute %r" % name) elif _is_magic(name): raise AttributeError(name) if not self._mock_unsafe: if name.startswith(('assert', 'assret')): > raise AttributeError(name) E AttributeError: assert_called_once /opt/python/3.5.6/lib/python3.5/unittest/mock.py:585: AttributeError This patch skips the tests if they are not in Python 3.6+. [1] https://travis-ci.org/starnight/pymodbus/jobs/584431003 [2] https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.assert_called_once Fixes: commit e6da559 asyncio server implementation (#400) * test: Make serve_forever() test only with Python 3.7+ If Python is older than 3.7, asyncio.base_events.Server.serve_forever will fail like PR riptideio#400 with "AttributeError: <class 'asyncio.base_events.Server'> does not have the attribute 'serve_forever'" [1] which is new in Python version 3.7 [2]: @asyncio.coroutine def testTcpServerServeNoDefer(self): ''' Test StartTcpServer without deferred start (immediate execution of server) ''' > with patch('asyncio.base_events.Server.serve_forever', new_callable=asynctest.CoroutineMock) as serve: test/test_server_asyncio.py:81: ... if not self.create and original is DEFAULT: raise AttributeError( > "%s does not have the attribute %r" % (target, name) ) E AttributeError: <class 'asyncio.base_events.Server'> does not have the attribute 'serve_forever' This patch skips the tests if they are not in Python 3.7+. [1] https://travis-ci.org/starnight/pymodbus/jobs/584212511 [2] https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Server.serve_forever Fixes: commit e6da559 asyncio server implementation (#400) * Add TLS feature for Modbus synchronous (#446) * Add TLS feature for Modbus synchronous Modbus.org released MODBUS/TCP Security Protocol Specification [1], which focuses variant of the Mobdbus/TCP protocol utilizing Transport Layer Security (TLS). This patch enables the Modbus over TLS feature as ModbusTlsClient with the Python builtin module ssl - TLS/SSL wrapper for socket objects. [1]: http://modbus.org/docs/MB-TCP-Security-v21_2018-07-24.pdf * Implement MODBUS TLS synchronous server Since we have the MODBUS TLS synchronous client, we can also have the MODBUS TLS synchronous server. * Fix #461 - Udp client/server , Fix #401 - package license with source, #457 Fix typo's in docstrings, #455-Support float16 * Fix examples, Merge #431 * #401 Move license to root folder from docs
dhoomakethu
added a commit
that referenced
this pull request
Sep 11, 2020
* Rebase to dev3.7 * Adding 3.7 to travis configuration * Updated documentation to resolve warnings introduced with the longer names Updated requirements-docs.txt to include missing modules * Fixed reference to deprecated asynchronous * Adding gmp disable to fix pypy build issues * Adding gmp disable to fix pypy build issues * Removing travis python 3.7 configuration Commenting out python3.7 from Travis while waiting for support. You can run teh 3.7 tests with tox without issues * Adding asserts for Payload Endianness * Fixing example of Payload. Same Endianness for builder and decoder. * Fix Sql db slave context validate and get methods - #139 * #353 - debugging, Add debug logs to check size of avaialble data in read buffer * #353 Provide an option to disable inter char timeouts * #353 Bump version, update changelog * check self.socket (#354) * check self.socket self.socket might be None at this point * Update pymodbus/client/sync.py Co-Authored-By: mpf82 <[email protected]> * Fix typo (#378) * Pymodbus 2.2.0 (#375) * #357 Support registration of custom requests * #368 Fixes write to broadcast address When writing to broadcast address (unit_id=0) there should be no response according to the Modbus spec. This fix changes expected_response_length to 0 when writing to unit_id=0. This will break any existing code that is improperly using unit_id 0 for a slave address. * Bump version to 2.2.0 Fix #366 Update failures in sql context Update Changelog Fix major minor version in example codes * Fix #371 pymodbus repl on python3 * 1. Fix tornado async serial client `TypeError` while processing incoming packet. 2. Fix asyncio examples. 3. Minor update in factory.py, now server logs prints received request instead of only function cod * [fix v3] poprawa sprawdzania timeout * Release candidate for pymodbus 2.2.0 * Fix #377 when invalid port is supplied and minor updates in logging * #368 adds broadcast support for sync client and server Adds broadcast_enable parameter to client and server, default value is False. When true it will treat unit_id 0 as broadcast and execute requests on all server slave contexts and not send a response and on the client side will send the request and not try to receive a response. * #368 Fixes minor bug in broadcast support code * Fixed erronous CRC handling If the CRC recieved is not correct in my case my slave got caught in a deadlock, not taking any new requests. This addition fixed that. * Update Changelog * Fix test coverage * Fix #387 Transactions failing on 2.2.0rc2. * Task Cancellation and CRC Errors Alternate solution for #356 and #360. Changes the RTU to make the transaction ID as the unit ID instead of an ever incrementing number. Previously this transaction ID was always 0 on the receiving end but was the unique transaction ID on sending. As such the FIFO buffer made the most sense. By tying it to the unit ID, we can recover from failure modes such as: - - Asyncio task cancellations (eg. timeouts) #360 - Skipped responses from slaves. (hangs on master #360) - CRC Errors #356 - Busy response * Cherry pick commit from PR #367 , Update changelog , bump version to 2.2.0rc4 * #389 Support passing all serial port parameters to asynchronous server * Fix BinaryPayloadDecoder and Builder wrt to coils * Misc updates, bump version to 2.2.0 * ReportSlaveIdResponse now tries to get slave id based on server identity for pymodbus servers * Update missing bcrypt requirement for testing * Fix docs (#407) * Fix document generation * Formatting updates in Changelog * Remove pycrypto dep (#411) It has not been needed by Twisted for a long time, and has been unmaintained for a long time. * Fix --upgrade option in install dependencies (#413) * Fix document generation * Formatting updates in Changelog * Fix --upgrade option in install dependencies * Padding for odd sized responses (#425) If the response is odd size the buffer needs to be padded with an additional byte. * README update: REPL stands for Read Evaluate **Print** Loop (#426) * Drop python 3.4 support (#440) Python 3.4 is EoL and has an easy upgrade path to 3.5+. Support was dropped in Twisted 19.7.0, which is causing Travis to fail pymodbus tests for 3.4. * Re-enable travis python 3.7 builds (#441) * Update __init__.py (#436) * Use SPDX identifier to specify the exact license type (#427) * asyncio server implementation (#400) * #357 Support registration of custom requests * #368 Fixes write to broadcast address When writing to broadcast address (unit_id=0) there should be no response according to the Modbus spec. This fix changes expected_response_length to 0 when writing to unit_id=0. This will break any existing code that is improperly using unit_id 0 for a slave address. * Bump version to 2.2.0 Fix #366 Update failures in sql context Update Changelog Fix major minor version in example codes * Fix #371 pymodbus repl on python3 * 1. Fix tornado async serial client `TypeError` while processing incoming packet. 2. Fix asyncio examples. 3. Minor update in factory.py, now server logs prints received request instead of only function cod * [fix v3] poprawa sprawdzania timeout * Release candidate for pymodbus 2.2.0 * Fix #377 when invalid port is supplied and minor updates in logging * #368 adds broadcast support for sync client and server Adds broadcast_enable parameter to client and server, default value is False. When true it will treat unit_id 0 as broadcast and execute requests on all server slave contexts and not send a response and on the client side will send the request and not try to receive a response. * #368 Fixes minor bug in broadcast support code * Fixed erronous CRC handling If the CRC recieved is not correct in my case my slave got caught in a deadlock, not taking any new requests. This addition fixed that. * Update Changelog * Fix test coverage * Fix #387 Transactions failing on 2.2.0rc2. * Task Cancellation and CRC Errors Alternate solution for #356 and #360. Changes the RTU to make the transaction ID as the unit ID instead of an ever incrementing number. Previously this transaction ID was always 0 on the receiving end but was the unique transaction ID on sending. As such the FIFO buffer made the most sense. By tying it to the unit ID, we can recover from failure modes such as: - - Asyncio task cancellations (eg. timeouts) #360 - Skipped responses from slaves. (hangs on master #360) - CRC Errors #356 - Busy response * Cherry pick commit from PR #367 , Update changelog , bump version to 2.2.0rc4 * native asyncio implementation of ModbusTcpServer and ModbusUdpServer * preliminary asyncio server examples * move serial module dependency into class instantiation * unittests for asyncio based server implementation * induce exception in execute method by mock patching the request object's execute method * move serial module dependency into class instantiation * added asynctest depency to requirements-tests.txt * add unittest skip condition for unsupported targets, remove failing assertion from unsupported targets, use lower asynctest version * remove logger setLevel call since doing so may override library consumers' already set log level * remove async def/await keywords from unittest so that the ast can be loaded in py2 even if the test is to be skipped * Add option to repl allowing Modbus RTU framing on a TCP socket (#447) * repl: Allow Modbus RTU framing on a TCP socket * repl: Update README for framing option * Fix asynci server test failures on python3.6 and below * Bump version to 2.2.0rc1, update six requirements and Changelog * Support multiple Python versions to fix test error from PR #400 (#444) * client/sync.py: Fix missing serial module dependency The serial.connect failed in PR riptideio#400 with "NameError: name 'serial' is not defined" [1]: self = <ModbusSerialClient at 0x7fcda4009b00 socket=None, method=ascii, timeout=3> def connect(self): """ Connect to the modbus serial server :returns: True if connection succeeded, False otherwise """ if self.socket: return True try: > self.socket = serial.Serial(port=self.port, timeout=self.timeout, bytesize=self.bytesize, stopbits=self.stopbits, baudrate=self.baudrate, parity=self.parity) E NameError: name 'serial' is not defined pymodbus/client/sync.py:476: NameError This patch moves the serial import back to the head. [1] https://travis-ci.org/riptideio/pymodbus/jobs/566009109 Fixes: commit e6da559 asyncio server implementation (#400) * server/asyncio.py: Create server with appropriate args and environment If Python is older than 3.7, the create_server will fail like PR riptideio#400 with "unexpected keyword argument 'start_serving'" [1] which is new in Python version 3.7: self.server_factory = self.loop.create_server(lambda :self.handler(self), *self.address, reuse_address=allow_reuse_address, reuse_port=allow_reuse_port, backlog=backlog, > start_serving=not defer_start) E TypeError: create_server() got an unexpected keyword argument 'start_serving' pymodbus/server/asyncio.py:400: TypeError This patch creates server according to Python environment. [1] https://travis-ci.org/starnight/pymodbus/jobs/584178484 Fixes: commit e6da559 asyncio server implementation (#400) * Create asyncio task with appropriate method and environment If Python is older than 3.7, the asyncio.create_task will fail like PR riptideio#400 with "AttributeError: module 'asyncio' has no attribute 'create_task'" [1] which is new in Python version 3.7 [2]: @asyncio.coroutine def testTcpServerCloseActiveConnection(self): ''' Test server_close() while there are active TCP connections''' data = b"\x01\x00\x00\x00\x00\x06\x01\x01\x00\x00\x00\x01" server = yield from StartTcpServer(context=self.context,address=("127.0.0.1", 0),loop=self.loop) > server_task = asyncio.create_task(server.serve_forever()) E AttributeError: module 'asyncio' has no attribute 'create_task' test/test_server_asyncio.py:205: AttributeError This patch creates task according to Python environment. [1] https://travis-ci.org/starnight/pymodbus/jobs/584193587 [2] https://docs.python.org/3/library/asyncio-task.html#creating-tasks Fixes: commit e6da559 asyncio server implementation (#400) * server/asyncio.py: Fix format string for older Python If Python is older than 3.6, f-Strings will fail like PR riptideio#400 with "SyntaxError: invalid syntax" [1] which is new in Python version 3.6 with PEP 498 -- Literal String Interpolation [2]: test/test_server_asyncio.py:14: in <module> from pymodbus.server.asyncio import StartTcpServer, StartUdpServer, StartSerialServer, StopServer, ModbusServerFactory E File "/home/travis/build/starnight/pymodbus/pymodbus/server/asyncio.py", line 424 E _logger.warning(f"aborting active session {k}") E ^ E SyntaxError: invalid syntax This patch fixes the format string with traditional format string syntax. [1] https://travis-ci.org/starnight/pymodbus/jobs/584427976 [2] https://www.python.org/dev/peps/pep-0498/ Fixes: commit e6da559 asyncio server implementation (#400) * test: Make assert_called_once() test only with Python 3.6+ If Python is older than 3.6, unittest.mock.assert_called_once() will fail like PR riptideio#400 with "AttributeError: assert_called_once" [1] which is new in Python version 3.6 [2]: > self.loop.create_server.assert_called_once() test/test_server_asyncio.py:76: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <CoroutineMock name='mock.create_server' id='139638313234784'> name = 'assert_called_once' def __getattr__(self, name): if name in {'_mock_methods', '_mock_unsafe'}: raise AttributeError(name) elif self._mock_methods is not None: if name not in self._mock_methods or name in _all_magics: raise AttributeError("Mock object has no attribute %r" % name) elif _is_magic(name): raise AttributeError(name) if not self._mock_unsafe: if name.startswith(('assert', 'assret')): > raise AttributeError(name) E AttributeError: assert_called_once /opt/python/3.5.6/lib/python3.5/unittest/mock.py:585: AttributeError This patch skips the tests if they are not in Python 3.6+. [1] https://travis-ci.org/starnight/pymodbus/jobs/584431003 [2] https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.assert_called_once Fixes: commit e6da559 asyncio server implementation (#400) * test: Make serve_forever() test only with Python 3.7+ If Python is older than 3.7, asyncio.base_events.Server.serve_forever will fail like PR riptideio#400 with "AttributeError: <class 'asyncio.base_events.Server'> does not have the attribute 'serve_forever'" [1] which is new in Python version 3.7 [2]: @asyncio.coroutine def testTcpServerServeNoDefer(self): ''' Test StartTcpServer without deferred start (immediate execution of server) ''' > with patch('asyncio.base_events.Server.serve_forever', new_callable=asynctest.CoroutineMock) as serve: test/test_server_asyncio.py:81: ... if not self.create and original is DEFAULT: raise AttributeError( > "%s does not have the attribute %r" % (target, name) ) E AttributeError: <class 'asyncio.base_events.Server'> does not have the attribute 'serve_forever' This patch skips the tests if they are not in Python 3.7+. [1] https://travis-ci.org/starnight/pymodbus/jobs/584212511 [2] https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Server.serve_forever Fixes: commit e6da559 asyncio server implementation (#400) * Add TLS feature for Modbus synchronous (#446) * Add TLS feature for Modbus synchronous Modbus.org released MODBUS/TCP Security Protocol Specification [1], which focuses variant of the Mobdbus/TCP protocol utilizing Transport Layer Security (TLS). This patch enables the Modbus over TLS feature as ModbusTlsClient with the Python builtin module ssl - TLS/SSL wrapper for socket objects. [1]: http://modbus.org/docs/MB-TCP-Security-v21_2018-07-24.pdf * Implement MODBUS TLS synchronous server Since we have the MODBUS TLS synchronous client, we can also have the MODBUS TLS synchronous server. * Fix #461 - Udp client/server , Fix #401 - package license with source, #457 Fix typo's in docstrings, #455-Support float16 * Fix examples, Merge #431 * #401 Move license to root folder from docs * rtu_framer: fix processing of incomplete frames (#466) * rtu_framer: fix processing of incomplete frames * rtu_framer: fix test case * Add handle local echo option * Update constants.py Added RetryOnInvalid flag and Backoff delay. * Update transaction.py Added retry on invalid data received and exponetial backoff delay between retries. * Add TLS feature for Modbus asynchronous (#470) * Add TLS feature for Modbus asynchronous client Since we have Modbus TLS client in synchronous mode, we can also implement Modbus TLS client in asynchronous mode with ASYNC_IO. * Add TLS feature for Modbus asynchronous server Since we have Modbus TLS server in synchronous mode, we can also implement Modbus TLS server in asynchronous mode with ASYNC_IO. * PR #471 Fix transaction tests * Fix failing tests * Add "Python" trove classifier Previously only generic "Python" support (without a version) was announced. * Merge PR's , bump version to 2.4.0 * closes #481, #482, #483, #484 * Closes #491 * Asyncio bug fixes (#517) * Closes #491 * 1. update requirements 2. Fix examples 3. Fix #494 - handle_local_echo 4. Fix #500 -- asyncio serial client with already running loop 5. Fix #486 - Pass serial args for asyncio serial client 6. Fix #490 - Typo in decode_data for socker_framer 7. Fix #385 - Support timeouts to break out of responspe await when server goes offline 8. Misc updates * #516 custom data block fix * Update Changelogs , bump version to 2.4.0 * #515 fix repl broadcast (#531) * 1. update requirements 2. Fix examples 3. Fix #494 - handle_local_echo 4. Fix #500 -- asyncio serial client with already running loop 5. Fix #486 - Pass serial args for asyncio serial client 6. Fix #490 - Typo in decode_data for socker_framer 7. Fix #385 - Support timeouts to break out of responspe await when server goes offline 8. Misc updates * #516 custom data block fix * Fix broadcast error with REPL client #515 * Fix #509 Wrong unit ID referenced in framers * Update documentation for serial forwarder example. Fixes #525 * Fix unit tests, support python 3.8 for tests, renamed: pymodbus/server/asyncio.py -> pymodbus/server/async_io.py and pymodbus/client/asynchronous/asyncio -> pymodbus/client/asynchronous/async_io * Ignore python3 code syntax while reporting coverage * Fix tests failing on python 3.6 and osx * Fix typo in makefile * Fix test execution errors specific to python3.6 * Osx travis issue - Fix trial 1 * Travis reverting xcode to 8.x for mac osx * Pymodbus v2.4.0 Co-authored-by: dices <[email protected]> Co-authored-by: Eric Duminil <[email protected]> Co-authored-by: Mike <[email protected]> Co-authored-by: Kim Hansen <[email protected]> Co-authored-by: Michael Corcoran <[email protected]> Co-authored-by: Andrea Canidio <[email protected]> Co-authored-by: tcplomp <[email protected]> Co-authored-by: alecjohanson <[email protected]> Co-authored-by: hackerboygn <[email protected]> Co-authored-by: Yegor Yefremov <[email protected]> Co-authored-by: Memet Bilgin <[email protected]> Co-authored-by: Sekenre <[email protected]> Co-authored-by: sanjay <[email protected]> Co-authored-by: Jian-Hong Pan <[email protected]> Co-authored-by: Steffen Vogel <[email protected]> Co-authored-by: Alexey Andreyev <[email protected]> Co-authored-by: Wild Stray <[email protected]> Co-authored-by: Lars Kruse <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a preliminary server implementation using core
asynciofunctionality.I have successfully run the examples/common/synchronous_client.py tests for TCP and UDP. I have not integrated other tests as of this commit.
Topics to be discussed:
handleimplementation for TCP and UDP. I'm not sure why those were and I've left a commented line here. Please comment.unittest: will require something likeasynctestor the equivalent. I can see there are some helper tests but haven't fully investigated. I'm willing to contribute some more but I need some guidance and I need to work on my main project at the same time. Especially useful would be if there are tests I can wholesale copy over.... any other comments?
(it goes without saying that this is not merge ready as is)
Thanks.