Add TLS feature for Modbus asynchronous #470
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since we have Modbus TLS client in synchronous mode, we can also implement Modbus TLS client in asynchronous mode with ASYNC_IO.
Since we have Modbus TLS server in synchronous mode, we can also implement Modbus TLS server in asynchronous mode with ASYNC_IO.
dhoomakethu
added a commit
that referenced
this pull request
Sep 11, 2020
* Rebase to dev3.7 * Adding 3.7 to travis configuration * Updated documentation to resolve warnings introduced with the longer names Updated requirements-docs.txt to include missing modules * Fixed reference to deprecated asynchronous * Adding gmp disable to fix pypy build issues * Adding gmp disable to fix pypy build issues * Removing travis python 3.7 configuration Commenting out python3.7 from Travis while waiting for support. You can run teh 3.7 tests with tox without issues * Adding asserts for Payload Endianness * Fixing example of Payload. Same Endianness for builder and decoder. * Fix Sql db slave context validate and get methods - #139 * #353 - debugging, Add debug logs to check size of avaialble data in read buffer * #353 Provide an option to disable inter char timeouts * #353 Bump version, update changelog * check self.socket (#354) * check self.socket self.socket might be None at this point * Update pymodbus/client/sync.py Co-Authored-By: mpf82 <[email protected]> * Fix typo (#378) * Pymodbus 2.2.0 (#375) * #357 Support registration of custom requests * #368 Fixes write to broadcast address When writing to broadcast address (unit_id=0) there should be no response according to the Modbus spec. This fix changes expected_response_length to 0 when writing to unit_id=0. This will break any existing code that is improperly using unit_id 0 for a slave address. * Bump version to 2.2.0 Fix #366 Update failures in sql context Update Changelog Fix major minor version in example codes * Fix #371 pymodbus repl on python3 * 1. Fix tornado async serial client `TypeError` while processing incoming packet. 2. Fix asyncio examples. 3. Minor update in factory.py, now server logs prints received request instead of only function cod * [fix v3] poprawa sprawdzania timeout * Release candidate for pymodbus 2.2.0 * Fix #377 when invalid port is supplied and minor updates in logging * #368 adds broadcast support for sync client and server Adds broadcast_enable parameter to client and server, default value is False. When true it will treat unit_id 0 as broadcast and execute requests on all server slave contexts and not send a response and on the client side will send the request and not try to receive a response. * #368 Fixes minor bug in broadcast support code * Fixed erronous CRC handling If the CRC recieved is not correct in my case my slave got caught in a deadlock, not taking any new requests. This addition fixed that. * Update Changelog * Fix test coverage * Fix #387 Transactions failing on 2.2.0rc2. * Task Cancellation and CRC Errors Alternate solution for #356 and #360. Changes the RTU to make the transaction ID as the unit ID instead of an ever incrementing number. Previously this transaction ID was always 0 on the receiving end but was the unique transaction ID on sending. As such the FIFO buffer made the most sense. By tying it to the unit ID, we can recover from failure modes such as: - - Asyncio task cancellations (eg. timeouts) #360 - Skipped responses from slaves. (hangs on master #360) - CRC Errors #356 - Busy response * Cherry pick commit from PR #367 , Update changelog , bump version to 2.2.0rc4 * #389 Support passing all serial port parameters to asynchronous server * Fix BinaryPayloadDecoder and Builder wrt to coils * Misc updates, bump version to 2.2.0 * ReportSlaveIdResponse now tries to get slave id based on server identity for pymodbus servers * Update missing bcrypt requirement for testing * Fix docs (#407) * Fix document generation * Formatting updates in Changelog * Remove pycrypto dep (#411) It has not been needed by Twisted for a long time, and has been unmaintained for a long time. * Fix --upgrade option in install dependencies (#413) * Fix document generation * Formatting updates in Changelog * Fix --upgrade option in install dependencies * Padding for odd sized responses (#425) If the response is odd size the buffer needs to be padded with an additional byte. * README update: REPL stands for Read Evaluate **Print** Loop (#426) * Drop python 3.4 support (#440) Python 3.4 is EoL and has an easy upgrade path to 3.5+. Support was dropped in Twisted 19.7.0, which is causing Travis to fail pymodbus tests for 3.4. * Re-enable travis python 3.7 builds (#441) * Update __init__.py (#436) * Use SPDX identifier to specify the exact license type (#427) * asyncio server implementation (#400) * #357 Support registration of custom requests * #368 Fixes write to broadcast address When writing to broadcast address (unit_id=0) there should be no response according to the Modbus spec. This fix changes expected_response_length to 0 when writing to unit_id=0. This will break any existing code that is improperly using unit_id 0 for a slave address. * Bump version to 2.2.0 Fix #366 Update failures in sql context Update Changelog Fix major minor version in example codes * Fix #371 pymodbus repl on python3 * 1. Fix tornado async serial client `TypeError` while processing incoming packet. 2. Fix asyncio examples. 3. Minor update in factory.py, now server logs prints received request instead of only function cod * [fix v3] poprawa sprawdzania timeout * Release candidate for pymodbus 2.2.0 * Fix #377 when invalid port is supplied and minor updates in logging * #368 adds broadcast support for sync client and server Adds broadcast_enable parameter to client and server, default value is False. When true it will treat unit_id 0 as broadcast and execute requests on all server slave contexts and not send a response and on the client side will send the request and not try to receive a response. * #368 Fixes minor bug in broadcast support code * Fixed erronous CRC handling If the CRC recieved is not correct in my case my slave got caught in a deadlock, not taking any new requests. This addition fixed that. * Update Changelog * Fix test coverage * Fix #387 Transactions failing on 2.2.0rc2. * Task Cancellation and CRC Errors Alternate solution for #356 and #360. Changes the RTU to make the transaction ID as the unit ID instead of an ever incrementing number. Previously this transaction ID was always 0 on the receiving end but was the unique transaction ID on sending. As such the FIFO buffer made the most sense. By tying it to the unit ID, we can recover from failure modes such as: - - Asyncio task cancellations (eg. timeouts) #360 - Skipped responses from slaves. (hangs on master #360) - CRC Errors #356 - Busy response * Cherry pick commit from PR #367 , Update changelog , bump version to 2.2.0rc4 * native asyncio implementation of ModbusTcpServer and ModbusUdpServer * preliminary asyncio server examples * move serial module dependency into class instantiation * unittests for asyncio based server implementation * induce exception in execute method by mock patching the request object's execute method * move serial module dependency into class instantiation * added asynctest depency to requirements-tests.txt * add unittest skip condition for unsupported targets, remove failing assertion from unsupported targets, use lower asynctest version * remove logger setLevel call since doing so may override library consumers' already set log level * remove async def/await keywords from unittest so that the ast can be loaded in py2 even if the test is to be skipped * Add option to repl allowing Modbus RTU framing on a TCP socket (#447) * repl: Allow Modbus RTU framing on a TCP socket * repl: Update README for framing option * Fix asynci server test failures on python3.6 and below * Bump version to 2.2.0rc1, update six requirements and Changelog * Support multiple Python versions to fix test error from PR #400 (#444) * client/sync.py: Fix missing serial module dependency The serial.connect failed in PR riptideio#400 with "NameError: name 'serial' is not defined" [1]: self = <ModbusSerialClient at 0x7fcda4009b00 socket=None, method=ascii, timeout=3> def connect(self): """ Connect to the modbus serial server :returns: True if connection succeeded, False otherwise """ if self.socket: return True try: > self.socket = serial.Serial(port=self.port, timeout=self.timeout, bytesize=self.bytesize, stopbits=self.stopbits, baudrate=self.baudrate, parity=self.parity) E NameError: name 'serial' is not defined pymodbus/client/sync.py:476: NameError This patch moves the serial import back to the head. [1] https://travis-ci.org/riptideio/pymodbus/jobs/566009109 Fixes: commit e6da559 asyncio server implementation (#400) * server/asyncio.py: Create server with appropriate args and environment If Python is older than 3.7, the create_server will fail like PR riptideio#400 with "unexpected keyword argument 'start_serving'" [1] which is new in Python version 3.7: self.server_factory = self.loop.create_server(lambda :self.handler(self), *self.address, reuse_address=allow_reuse_address, reuse_port=allow_reuse_port, backlog=backlog, > start_serving=not defer_start) E TypeError: create_server() got an unexpected keyword argument 'start_serving' pymodbus/server/asyncio.py:400: TypeError This patch creates server according to Python environment. [1] https://travis-ci.org/starnight/pymodbus/jobs/584178484 Fixes: commit e6da559 asyncio server implementation (#400) * Create asyncio task with appropriate method and environment If Python is older than 3.7, the asyncio.create_task will fail like PR riptideio#400 with "AttributeError: module 'asyncio' has no attribute 'create_task'" [1] which is new in Python version 3.7 [2]: @asyncio.coroutine def testTcpServerCloseActiveConnection(self): ''' Test server_close() while there are active TCP connections''' data = b"\x01\x00\x00\x00\x00\x06\x01\x01\x00\x00\x00\x01" server = yield from StartTcpServer(context=self.context,address=("127.0.0.1", 0),loop=self.loop) > server_task = asyncio.create_task(server.serve_forever()) E AttributeError: module 'asyncio' has no attribute 'create_task' test/test_server_asyncio.py:205: AttributeError This patch creates task according to Python environment. [1] https://travis-ci.org/starnight/pymodbus/jobs/584193587 [2] https://docs.python.org/3/library/asyncio-task.html#creating-tasks Fixes: commit e6da559 asyncio server implementation (#400) * server/asyncio.py: Fix format string for older Python If Python is older than 3.6, f-Strings will fail like PR riptideio#400 with "SyntaxError: invalid syntax" [1] which is new in Python version 3.6 with PEP 498 -- Literal String Interpolation [2]: test/test_server_asyncio.py:14: in <module> from pymodbus.server.asyncio import StartTcpServer, StartUdpServer, StartSerialServer, StopServer, ModbusServerFactory E File "/home/travis/build/starnight/pymodbus/pymodbus/server/asyncio.py", line 424 E _logger.warning(f"aborting active session {k}") E ^ E SyntaxError: invalid syntax This patch fixes the format string with traditional format string syntax. [1] https://travis-ci.org/starnight/pymodbus/jobs/584427976 [2] https://www.python.org/dev/peps/pep-0498/ Fixes: commit e6da559 asyncio server implementation (#400) * test: Make assert_called_once() test only with Python 3.6+ If Python is older than 3.6, unittest.mock.assert_called_once() will fail like PR riptideio#400 with "AttributeError: assert_called_once" [1] which is new in Python version 3.6 [2]: > self.loop.create_server.assert_called_once() test/test_server_asyncio.py:76: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <CoroutineMock name='mock.create_server' id='139638313234784'> name = 'assert_called_once' def __getattr__(self, name): if name in {'_mock_methods', '_mock_unsafe'}: raise AttributeError(name) elif self._mock_methods is not None: if name not in self._mock_methods or name in _all_magics: raise AttributeError("Mock object has no attribute %r" % name) elif _is_magic(name): raise AttributeError(name) if not self._mock_unsafe: if name.startswith(('assert', 'assret')): > raise AttributeError(name) E AttributeError: assert_called_once /opt/python/3.5.6/lib/python3.5/unittest/mock.py:585: AttributeError This patch skips the tests if they are not in Python 3.6+. [1] https://travis-ci.org/starnight/pymodbus/jobs/584431003 [2] https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.assert_called_once Fixes: commit e6da559 asyncio server implementation (#400) * test: Make serve_forever() test only with Python 3.7+ If Python is older than 3.7, asyncio.base_events.Server.serve_forever will fail like PR riptideio#400 with "AttributeError: <class 'asyncio.base_events.Server'> does not have the attribute 'serve_forever'" [1] which is new in Python version 3.7 [2]: @asyncio.coroutine def testTcpServerServeNoDefer(self): ''' Test StartTcpServer without deferred start (immediate execution of server) ''' > with patch('asyncio.base_events.Server.serve_forever', new_callable=asynctest.CoroutineMock) as serve: test/test_server_asyncio.py:81: ... if not self.create and original is DEFAULT: raise AttributeError( > "%s does not have the attribute %r" % (target, name) ) E AttributeError: <class 'asyncio.base_events.Server'> does not have the attribute 'serve_forever' This patch skips the tests if they are not in Python 3.7+. [1] https://travis-ci.org/starnight/pymodbus/jobs/584212511 [2] https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Server.serve_forever Fixes: commit e6da559 asyncio server implementation (#400) * Add TLS feature for Modbus synchronous (#446) * Add TLS feature for Modbus synchronous Modbus.org released MODBUS/TCP Security Protocol Specification [1], which focuses variant of the Mobdbus/TCP protocol utilizing Transport Layer Security (TLS). This patch enables the Modbus over TLS feature as ModbusTlsClient with the Python builtin module ssl - TLS/SSL wrapper for socket objects. [1]: http://modbus.org/docs/MB-TCP-Security-v21_2018-07-24.pdf * Implement MODBUS TLS synchronous server Since we have the MODBUS TLS synchronous client, we can also have the MODBUS TLS synchronous server. * Fix #461 - Udp client/server , Fix #401 - package license with source, #457 Fix typo's in docstrings, #455-Support float16 * Fix examples, Merge #431 * #401 Move license to root folder from docs * rtu_framer: fix processing of incomplete frames (#466) * rtu_framer: fix processing of incomplete frames * rtu_framer: fix test case * Add handle local echo option * Update constants.py Added RetryOnInvalid flag and Backoff delay. * Update transaction.py Added retry on invalid data received and exponetial backoff delay between retries. * Add TLS feature for Modbus asynchronous (#470) * Add TLS feature for Modbus asynchronous client Since we have Modbus TLS client in synchronous mode, we can also implement Modbus TLS client in asynchronous mode with ASYNC_IO. * Add TLS feature for Modbus asynchronous server Since we have Modbus TLS server in synchronous mode, we can also implement Modbus TLS server in asynchronous mode with ASYNC_IO. * PR #471 Fix transaction tests * Fix failing tests * Add "Python" trove classifier Previously only generic "Python" support (without a version) was announced. * Merge PR's , bump version to 2.4.0 * closes #481, #482, #483, #484 * Closes #491 * Asyncio bug fixes (#517) * Closes #491 * 1. update requirements 2. Fix examples 3. Fix #494 - handle_local_echo 4. Fix #500 -- asyncio serial client with already running loop 5. Fix #486 - Pass serial args for asyncio serial client 6. Fix #490 - Typo in decode_data for socker_framer 7. Fix #385 - Support timeouts to break out of responspe await when server goes offline 8. Misc updates * #516 custom data block fix * Update Changelogs , bump version to 2.4.0 * #515 fix repl broadcast (#531) * 1. update requirements 2. Fix examples 3. Fix #494 - handle_local_echo 4. Fix #500 -- asyncio serial client with already running loop 5. Fix #486 - Pass serial args for asyncio serial client 6. Fix #490 - Typo in decode_data for socker_framer 7. Fix #385 - Support timeouts to break out of responspe await when server goes offline 8. Misc updates * #516 custom data block fix * Fix broadcast error with REPL client #515 * Fix #509 Wrong unit ID referenced in framers * Update documentation for serial forwarder example. Fixes #525 * Fix unit tests, support python 3.8 for tests, renamed: pymodbus/server/asyncio.py -> pymodbus/server/async_io.py and pymodbus/client/asynchronous/asyncio -> pymodbus/client/asynchronous/async_io * Ignore python3 code syntax while reporting coverage * Fix tests failing on python 3.6 and osx * Fix typo in makefile * Fix test execution errors specific to python3.6 * Osx travis issue - Fix trial 1 * Travis reverting xcode to 8.x for mac osx * Pymodbus v2.4.0 Co-authored-by: dices <[email protected]> Co-authored-by: Eric Duminil <[email protected]> Co-authored-by: Mike <[email protected]> Co-authored-by: Kim Hansen <[email protected]> Co-authored-by: Michael Corcoran <[email protected]> Co-authored-by: Andrea Canidio <[email protected]> Co-authored-by: tcplomp <[email protected]> Co-authored-by: alecjohanson <[email protected]> Co-authored-by: hackerboygn <[email protected]> Co-authored-by: Yegor Yefremov <[email protected]> Co-authored-by: Memet Bilgin <[email protected]> Co-authored-by: Sekenre <[email protected]> Co-authored-by: sanjay <[email protected]> Co-authored-by: Jian-Hong Pan <[email protected]> Co-authored-by: Steffen Vogel <[email protected]> Co-authored-by: Alexey Andreyev <[email protected]> Co-authored-by: Wild Stray <[email protected]> Co-authored-by: Lars Kruse <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since we have Modbus TLS client/server in synchronous mode #446, we can also implement Modbus TLS client/server in asynchronous mode with ASYNC_IO. These commits include the implementation, examples and tests.