Enhance robustness and add log rotation support for mux simulator

ansible/roles/vm_set/files/mux_simulator.md

@@ -62,6 +62,28 @@ sudo systemctl restart mux-simulator
 The mux-simulator service is shared by multiple dualtor test setups using the same test server. Any dualtor test setups using it is recorded in a persistent file on test server `{{ root_path }}/mux_simulator.setups.txt`. During `testbed-cli.sh add-topo`, the vm set name of current setup will be added into it. During `testbed-cli.sh remove-topo`, the vm set name of current setup will be removed from it. When the file is empty, the mux-simulator service will be stopped.
 
 
+## How to troubleshoot mux simulator
+By default, the mux-simulator service output its logs to `/tmp/mux_simulator.log`. Default debug level is INFO. If DEBUG level logging is needed for troubleshooting, please follow below steps:
+
+1. Stop the mux-simulator service.
+```
+sudo systemctl stop mux-simulator
+```
+2. Find out path of the mux_simulator.py script from the mux-simulator systemd service file.
+```
+cat /etc/systemd/system/mux-simulator.service
+```
+3. Manually run the mux_simulator.py script with `-v` option to **turn on DEBUG level logging**.
+```
+ sudo /usr/bin/env python /home/azure/veos-vm/mux_simulator.py 8080 -v
+```
+4. Try to call the mux simulator HTTP APIs and check the log file `/tmp/mux_simulator.log` for detailed logging.
+5. After troubleshooting is done, stop the manually started mux_simulator.py script (for example: Ctrl+C).
+6. Start the mux-simulator service again.
+```
+sudo systemctl start mux-simulator
+```
+
 ## APIs
 The APIs using json for data exchange.
 

ansible/roles/vm_set/files/mux_simulator.py

@@ -15,9 +15,11 @@
 import subprocess
 import sys
 
+from logging.handlers import RotatingFileHandler
 from collections import defaultdict
 
 from flask import Flask, request, jsonify
+from flask.logging import default_handler
 
 app = Flask(__name__)
 
@@ -27,12 +29,6 @@
 NIC = 'nic'
 
 
-logging.basicConfig(
-    filename='/tmp/mux_simulator.log',
-    level=logging.INFO,
-    format='%(asctime)s %(levelname)s %(message)s')
-
-
 def run_cmd(cmdline):
     """Use subprocess to run a command line with shell=True
 
@@ -402,8 +398,13 @@ def get_mux_bridges(vm_set):
     """
     bridge_prefix = 'mbr-{}-'.format(vm_set)
     mux_bridges = [intf for intf in os.listdir('/sys/class/net') if intf.startswith(bridge_prefix)]
+    valid_mux_bridges = []
+    for mux_bridge in mux_bridges:
+        out = run_cmd('ovs-vsctl list-ports {}'.format(mux_bridge))
+        if len(out.splitlines()) ==3:
+            valid_mux_bridges.append(mux_bridge)
 
-    return mux_bridges
+    return valid_mux_bridges
 
 
 def get_all_mux_status(vm_set):
@@ -606,13 +607,29 @@ def mux_cable_flow_update(vm_set, port_index, action):
         return jsonify({'err_msg': err_msg}), 500
 
 
+def config_logging():
+    rfh = RotatingFileHandler(
+        '/tmp/mux_simulator.log',
+        maxBytes=1024*1024,
+        backupCount=5)
+    fmt = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
+    rfh.setFormatter(fmt)
+    rfh.setLevel(logging.INFO)
+    app.logger.addHandler(rfh)
+    app.logger.removeHandler(default_handler)
+
+
 if __name__ == '__main__':
     usage = '''
     Start mux simulator server at specified port.
     $ sudo python <prog> <port>
     '''
+    config_logging()
+
     if '-v' in sys.argv:
         app.logger.setLevel(logging.DEBUG)
+        for handler in app.logger.handlers:
+            handler.setLevel(logging.DEBUG)
 
     if len(sys.argv) < 2:
         app.logger.error(usage)