Test dev (#240)

* PYM-2:
- The issue listed was due to wrong messages being passed by the user.
Upon passing the right messages codes, the parser works as expected
on all counts.

- There is also changes that make the parser compatible with python2
and python3

- Verifier that the tool works on both python3 and python2 for all
MODBUS message codes on TCP, RTU, and BIN

* PYM-2: Checking the incoming framer. If te Framer is a Binary Framer,
we take the unit address as the second incoming bite as opposed
to the first bite.
This is was done while fixing the message parsers for binary messages

* PYM-2: Changed the modbus binary header size from 2 to 1.
According to the docs:
    Modbus Binary Frame Controller::

        [ Start ][Address ][ Function ][ Data ][ CRC ][ End ]
          1b        1b         1b         Nb     2b     1b

* PYM-3: Script is now compatible with both python2 and python3

* WIP

* PYM-2: Added a new switch: -t --transaction
This switch is meant to be used when we wish to parse messages directly
from the logs of Modbus.
The format of a message as shown in the logs is like bellow:
        0x7b 0x1 0x5 0x0 0x0 0xff 0x0 0x8c 0x3a 0x7d
We can pass this as the message to the parser along with the -t witch
to convert it into a compatible message to be parsed.
EG:

(modbus3) [~/pymodbus/examples/contrib]$ ./message-parser.py -b -t -p binary -m "0x7b 0x1 0x5 0x0 0x0 0xff 0x0 0x8c 0x3a 0x7d"
================================================================================
Decoding Message b'7b01050000ff008c3a7d'
================================================================================
ServerDecoder
--------------------------------------------------------------------------------
name            = WriteSingleCoilRequest
transaction_id  = 0x0
protocol_id     = 0x0
unit_id         =
                . [1]
skip_encode     = 0x0
check           = 0x0
address         = 0x0
value           = 0x1
documentation   =
    This function code is used to write a single output to either ON or OFF
    in a remote device.

    The requested ON/OFF state is specified by a constant in the request
    data field. A value of FF 00 hex requests the output to be ON. A value
    of 00 00 requests it to be OFF. All other values are illegal and will
    not affect the output.

    The Request PDU specifies the address of the coil to be forced. Coils
    are addressed starting at zero. Therefore coil numbered 1 is addressed
    as 0. The requested ON/OFF state is specified by a constant in the Coil
    Value field. A value of 0XFF00 requests the coil to be ON. A value of
    0X0000 requests the coil to be off. All other values are illegal and
    will not affect the coil.

ClientDecoder
--------------------------------------------------------------------------------
name            = WriteSingleCoilResponse
transaction_id  = 0x0
protocol_id     = 0x0
unit_id         =
                . [1]
skip_encode     = 0x0
check           = 0x0
address         = 0x0
value           = 0x1
documentation   =
    The normal response is an echo of the request, returned after the coil
    state has been written.

* PYM-2: Removing additional dependancy and making use of existing porting tools

* PYM-3: Removing additional dependancy and making use of existing porting tools

* Initial Bitbucket Pipelines configuration

* bitbucket-pipelines.yml edited online with Bitbucket

* bitbucket-pipelines.yml edited online with Bitbucket

* PYM-2: Updated the transaction tests for BinaryFramerTransaction.
The header for Binary trasaction is of size 1. This was recrtified
earlier commits of this branch.
The test ensure these changes

* PYM-6: Minor Cleanup task

Removing the argument handler in TCP Syncronous server.
This argument is not used any where.

* PYM-6:
ModbusUdpServer and ModbusTcpServer will now accept any legal
Modbus request handler. The request handler being passed will have
to be of instance ModbusBaseRequestHandler.

The default request handler is ModbusDisconnectedRequestHandler.
I.e., is no handler is passed, or if the handler is not of type
ModbusBaseRequestHandler, ModbusDisconnectedRequestHandler will
be made use of.

* PYM-6: Removing uneccessary check if handler is of type ModbusBaseRequestHandler

* PYM-8: Example that read from a database as a datastore

* PYM-8: Added two new datastores that can be used.
- SQLite3
- Reddis

* Small fixes

* Small fixes

* Small fixes

* Cleanup

* PYM-8: Updated the example to first write a random value at a random afddress to a database and then read from that address

* PYM-8: Added neccessary checks and methods to allow hassle free writes to database. The process first checks if the address and value are already present in the database before performing a write. This ensures that database transaction errors will now occur in cases where repetetive data is being written.

* Cleanup: Removing pdb placed during testing and other comments

* bitbucket-pipelines.yml deleted online with Bitbucket

* #240 Fix PR failures

* #240 fix Travis build failures

* #190 fix import error in dbstore-update-server example

* Small changes and typo fixed in pymodbus utilities and redis datastore helpers

* Added tests for redis datastore helpers

* Minor fixes to SQL datastore

* Unit tests for SQL datastore - 100% coverage

* Tests now compatible with python3 and python2

examples/common/dbstore-update-server.py

@@ -0,0 +1,94 @@
+'''
+Pymodbus Server With Updating Thread
+--------------------------------------------------------------------------
+This is an example of having a background thread updating the
+context in an SQLite4 database while the server is operating.
+
+This scrit generates a random address range (within 0 - 65000) and a random
+value and stores it in a database. It then reads the same address to verify
+that the process works as expected
+
+This can also be done with a python thread::
+    from threading import Thread
+    thread = Thread(target=updating_writer, args=(context,))
+    thread.start()
+'''
+#---------------------------------------------------------------------------#
+# import the modbus libraries we need
+#---------------------------------------------------------------------------#
+from pymodbus.server.async import StartTcpServer
+from pymodbus.device import ModbusDeviceIdentification
+from pymodbus.datastore import ModbusSequentialDataBlock
+from pymodbus.datastore import ModbusServerContext
+from pymodbus.datastore.database import SqlSlaveContext
+from pymodbus.transaction import ModbusRtuFramer, ModbusAsciiFramer
+import random
+
+#---------------------------------------------------------------------------#
+# import the twisted libraries we need
+#---------------------------------------------------------------------------#
+from twisted.internet.task import LoopingCall
+
+#---------------------------------------------------------------------------#
+# configure the service logging
+#---------------------------------------------------------------------------#
+import logging
+logging.basicConfig()
+log = logging.getLogger()
+log.setLevel(logging.DEBUG)
+
+#---------------------------------------------------------------------------#
+# define your callback process
+#---------------------------------------------------------------------------#
+def updating_writer(a):
+    ''' A worker process that runs every so often and
+    updates live values of the context which resides in an SQLite3 database.
+    It should be noted that there is a race condition for the update.
+    :param arguments: The input arguments to the call
+    '''
+    log.debug("Updating the database context")
+    context  = a[0]
+    readfunction = 0x03 # read holding registers
+    writefunction = 0x10
+    slave_id = 0x01 # slave address
+    count = 50
+
+    # import pdb; pdb.set_trace()
+
+    rand_value = random.randint(0, 9999)
+    rand_addr = random.randint(0, 65000)
+    log.debug("Writing to datastore: {}, {}".format(rand_addr, rand_value))
+    # import pdb; pdb.set_trace()
+    context[slave_id].setValues(writefunction, rand_addr, [rand_value])
+    values = context[slave_id].getValues(readfunction, rand_addr, count)
+    log.debug("Values from datastore: " + str(values))
+
+
+
+#---------------------------------------------------------------------------#
+# initialize your data store
+#---------------------------------------------------------------------------#
+block = ModbusSequentialDataBlock(0x00, [0]*0xff)
+store = SqlSlaveContext(block)
+
+context = ModbusServerContext(slaves={1: store}, single=False)
+
+
+#---------------------------------------------------------------------------#
+# initialize the server information
+#---------------------------------------------------------------------------#
+identity = ModbusDeviceIdentification()
+identity.VendorName  = 'pymodbus'
+identity.ProductCode = 'PM'
+identity.VendorUrl   = 'http://github.com/bashwork/pymodbus/'
+identity.ProductName = 'pymodbus Server'
+identity.ModelName   = 'pymodbus Server'
+identity.MajorMinorRevision = '1.0'
+
+#---------------------------------------------------------------------------#
+# run the server you want
+#---------------------------------------------------------------------------#
+time = 5 # 5 seconds delay
+loop = LoopingCall(f=updating_writer, a=(context,))
+loop.start(time, now=False) # initially delay by time
+StartTcpServer(context, identity=identity, address=("", 5020))

examples/contrib/message-generator.py

@@ -12,6 +12,7 @@
 * binary - `./generate-messages.py -f binary -m tx -b`
 '''
 from optparse import OptionParser
+import codecs as c
 #--------------------------------------------------------------------------#
 # import all the available framers
 #--------------------------------------------------------------------------#
@@ -30,6 +31,7 @@
 from pymodbus.mei_message import *
 from pymodbus.register_read_message import *
 from pymodbus.register_write_message import *
+from pymodbus.compat import IS_PYTHON3
 
 #--------------------------------------------------------------------------#
 # initialize logging
@@ -51,17 +53,17 @@
     WriteSingleRegisterRequest,
     WriteSingleCoilRequest,
     ReadWriteMultipleRegistersRequest,
-    
+
     ReadExceptionStatusRequest,
     GetCommEventCounterRequest,
     GetCommEventLogRequest,
     ReportSlaveIdRequest,
-    
+
     ReadFileRecordRequest,
     WriteFileRecordRequest,
     MaskWriteRegisterRequest,
     ReadFifoQueueRequest,
-    
+
     ReadDeviceInformationRequest,
 
     ReturnQueryDataRequest,
@@ -97,7 +99,7 @@
     WriteSingleRegisterResponse,
     WriteSingleCoilResponse,
     ReadWriteMultipleRegistersResponse,
-    
+
     ReadExceptionStatusResponse,
     GetCommEventCounterResponse,
     GetCommEventLogResponse,
@@ -149,13 +151,13 @@
     'write_registers'   : [0x01] * 8,
     'transaction'       : 0x01,
     'protocol'          : 0x00,
-    'unit'              : 0x01,
+    'unit'              : 0xff,
 }
 
 
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 # generate all the requested messages
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 def generate_messages(framer, options):
     ''' A helper method to parse the command line options
 
@@ -168,13 +170,16 @@ def generate_messages(framer, options):
         print ("%-44s = " % message.__class__.__name__)
         packet = framer.buildPacket(message)
         if not options.ascii:
-            packet = packet.encode('hex') + '\n'
-        print (packet)   # because ascii ends with a \r\n
+            if not IS_PYTHON3:
+                packet = packet.encode('hex')
+            else:
+                packet = c.encode(packet, 'hex_codec').decode('utf-8')
+        print ("{}\n".format(packet))   # because ascii ends with a \r\n
 
 
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 # initialize our program settings
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 def get_options():
     ''' A helper method to parse the command line options
 

examples/contrib/message-parser.py

@@ -11,20 +11,24 @@
 * rtu
 * binary
 '''
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 # import needed libraries
 #---------------------------------------------------------------------------#
 from __future__ import print_function
 import sys
 import collections
 import textwrap
 from optparse import OptionParser
+import codecs as c
+
 from pymodbus.utilities import computeCRC, computeLRC
 from pymodbus.factory import ClientDecoder, ServerDecoder
 from pymodbus.transaction import ModbusSocketFramer
 from pymodbus.transaction import ModbusBinaryFramer
 from pymodbus.transaction import ModbusAsciiFramer
 from pymodbus.transaction import ModbusRtuFramer
+from pymodbus.compat import byte2int, int2byte, IS_PYTHON3
+
 
 #--------------------------------------------------------------------------#
 # Logging
@@ -33,9 +37,9 @@
 modbus_log = logging.getLogger("pymodbus")
 
 
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 # build a quick wrapper around the framers
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 class Decoder(object):
 
     def __init__(self, framer, encode=False):
@@ -52,7 +56,10 @@ def decode(self, message):
 
         :param message: The messge to decode
         '''
-        value = message if self.encode else message.encode('hex')
+        if IS_PYTHON3:
+            value = message if self.encode else c.encode(message, 'hex_codec')
+        else:
+            value = message if self.encode else message.encode('hex')
         print("="*80)
         print("Decoding Message %s" % value)
         print("="*80)
@@ -64,7 +71,7 @@ def decode(self, message):
             print("%s" % decoder.decoder.__class__.__name__)
             print("-"*80)
             try:
-                decoder.addToFrame(message.encode())
+                decoder.addToFrame(message)
                 if decoder.checkFrame():
                     decoder.advanceFrame()
                     decoder.processIncomingPacket(message, self.report)
@@ -86,7 +93,7 @@ def report(self, message):
         :param message: The message to print
         '''
         print("%-15s = %s" % ('name', message.__class__.__name__))
-        for k,v in message.__dict__.iteritems():
+        for (k, v) in message.__dict__.items():
             if isinstance(v, dict):
                 print("%-15s =" % k)
                 for kk,vv in v.items():
@@ -102,9 +109,9 @@ def report(self, message):
         print("%-15s = %s" % ('documentation', message.__doc__))
 
 
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 # and decode our message
-#---------------------------------------------------------------------------# 
+#---------------------------------------------------------------------------#
 def get_options():
     ''' A helper method to parse the command line options
 
@@ -136,6 +143,10 @@ def get_options():
         help="The file containing messages to parse",
         dest="file", default=None)
 
+    parser.add_option("-t", "--transaction",
+        help="If the incoming message is in hexadecimal format",
+        action="store_true", dest="transaction", default=False)
+
     (opt, arg) = parser.parse_args()
 
     if not opt.message and len(arg) > 0:
@@ -150,8 +161,19 @@ def get_messages(option):
     :returns: The message iterator to parse
     '''
     if option.message:
+        if option.transaction:
+            msg = ""
+            for segment in option.message.split():
+                segment = segment.replace("0x", "")
+                segment = "0" + segment if len(segment) == 1 else segment
+                msg = msg + segment
+            option.message = msg
+
         if not option.ascii:
-            option.message = option.message.decode('hex')
+            if not IS_PYTHON3:
+                option.message = option.message.decode('hex')
+            else:
+                option.message = c.decode(option.message.encode(), 'hex_codec')
         yield option.message
     elif option.file:
         with open(option.file, "r") as handle:

pymodbus/datastore/database/__init__.py

@@ -0,0 +1,7 @@
+from pymodbus.datastore.database.sql_datastore import SqlSlaveContext
+from pymodbus.datastore.database.redis_datastore import RedisSlaveContext
+
+#---------------------------------------------------------------------------#
+# Exported symbols
+#---------------------------------------------------------------------------#
+__all__ = ["SqlSlaveContext", "RedisSlaveContext"]