feat(cli): fully support plugin parsers and connectors via cli

adds arguments to specify additional modules to include from the cli, adds documentation on using plugins via cli, and adjusts some logging and tools to make debugging and troubleshooting easier in the future.

docs/DOCUMENTATION.md

@@ -484,12 +484,14 @@ configuration file.
 
 ## Arguments
 
-| Argument               | Required | Description                                                        |
-|------------------------|:---------|--------------------------------------------------------------------|
-| `--config (-c) <FILE>` | Yes      | Specifies a config file to load. This must be a JSON or YAML file. |
-| `--version (-V)`       | No       | Prints the version of mail2beyond installed.                       |
-| `--verbose (-v)`       | No       | Enables verbose logging.                                           |
-| `--help (-h)`          | No       | Prints the help page.                                              |
+| Argument                 | Required | Description                                                          |
+|--------------------------|:---------|----------------------------------------------------------------------|
+| `--config (-c) <FILE>`   | Yes      | Specifies a config file to load. This must be a JSON or YAML file.   |
+| `--connector-plugin-dir` | No       | Specifies a path to a directory containing plugin connector modules. |
+| `--parser-plugin-dir`    | No       | Specifies a path to a directory containing plugin parser modules.    |
+| `--version (-V)`         | No       | Prints the version of mail2beyond installed.                         |
+| `--verbose (-v)`         | No       | Enables verbose logging.                                             |
+| `--help (-h)`            | No       | Prints the help page.                                                |
 
 ## Configuration
 A configuration file must be written before ```mail2beyond``` is started. It may also be helpful to check out the 
@@ -572,7 +574,8 @@ configuration. This name will be used to assign this connector to mappings in yo
 
 - _Required_: Yes
 - _Options_: [`void`, `smtp`, `slack`, `discord`, `google_chat`, `microsoft_teams`]
-- _Description_: The module this connector will use. Multiple connectors can use the same underlying module.
+- _Description_: The module this connector will use. Multiple connectors can use the same underlying module. You can 
+also [use your own custom connector modules](#using-plugin-connector-and-parser-modules).
 
 **config**
 
@@ -605,10 +608,9 @@ configuration. Multiple mappings can use the same the connector.
 - _Required_: No
 - _Options_: [`auto`, `plain`, `html`]
 - _Default_: `auto`
-- _Description_: Explicitly set the content-type parser to use. By default, the `auto` parser will be chosen to 
-select the parser that best matches the SMTP message's content-type header. The `plain` parser will not parse the
-content body and is the fallback for the `auto` parser if no parser exists for the content-type. The `html` parser
-will parse the content body as HTML and convert it to a more human-readable markdown format.
+- _Description_: Explicitly set the content-type parser to use. See the [built-in parsers section](#built-in-parsers)
+for more information on available parsers. You can also 
+[use your own custom parser modules](#using-plugin-connector-and-parser-modules).
 - _Notes_:
   - Some connector modules do not respect the `parser` option (e.g. `smtp`, `void`)
   - Even though the `html` parser converts the content to markdown, this does not guarantee the markdown content will
@@ -627,6 +629,63 @@ Once you have your configuration file written, you can start the server by runni
 mail2beyond --config /path/to/your/config.yml
 ```
 
+## Using Plugin Connector and Parser Modules
+If you have written your own [connector modules](#writing-custom-connectors) and/or 
+[parser modules](#writing-custom-parsers), you can include them from the CLI using the `--connector-plugins-dir` and 
+`--parser-plugins-dir` commands respectively. If specified, both these commands require a path to an existing directory 
+where your plugin connector and parser modules are stored. In order for plugins to be included, your modules must use 
+the `.py` extension within the specified directory. Additionally, extended `BaseConnector` classes must be named 
+`Connector` and extended `BaseParser` classes must be named `Parser` in order for them to be used via CLI. 
+
+In example, say we have a file structure like this:
+
+```
+plugins/
+├─ connectors/
+│  ├─ my_custom_connector_module_1.py
+│  ├─ my_custom_connector_module_2.py
+├─ parsers/
+│  ├─ my_custom_parser_1.py
+```
+
+In your configuration, you can utilize the custom connectors and parsers like such:
+
+```yaml
+---
+listeners:
+  - address: localhost
+    port: 25
+
+mappings:
+  - pattern: default
+    field: to
+    parser: my_custom_parser_1
+    connector: custom_connector_1
+
+  - pattern: "some custom pattern"
+    field: to
+    parser: my_custom_parser_1
+    connector: custom_connector_2
+
+connectors:
+  - name: custom_connector_1
+    module: my_custom_connector_module_1
+
+  - name: custom_connector_2
+    module: my_custom_connector_module_2
+```
+
+And finally, you can start the server and set the plugin module directories:
+```commandline
+mail2beyond --config /path/to/your/config.yml --connector-plugins-dir plugins/connectors/ --parser-plugins-dir plugins/parsers/ 
+```
+
+**Notes**
+
+- Even when you specify plugin modules, you will still have access to all the built-in connectors and parsers. In the 
+  event that your custom module uses the same name as a built-in module, _your module takes priority over the built-in
+  parser and the built-in parser will not be available to use.
+
 # Built-in Connectors
 Connector modules may contain their own configurable options and requirements. Below are the available configuration
 options available to each built-in connector module:

mail2beyond/framework.py

@@ -127,6 +127,7 @@ def start(self):
         """
         self.controller.start()
         self.log.info(f"mail2beyond started listening on {self.address}:{self.port}")
+        self.log_mappings()
 
     @staticmethod
     def wait():
@@ -204,6 +205,16 @@ def setup_logging(self, level: int = logging.NOTSET, handler=None, **kwargs):
         if level == logging.DEBUG:
             self.log.warning("logging at level DEBUG may expose sensitive information in logs")
 
+    def log_mappings(self):
+        """Logs the configured mappings upon startup for debugging."""
+        # Loop through all configured mappings and log it's configuration.
+        for mapping in self.mappings:
+            self.log.debug(
+                f"loaded mapping with pattern='{ mapping.pattern }' field='{ mapping.field }'"
+                f" connector='{ mapping.connector }' connector_module='{ mapping.connector.__module__ }'"
+                f" parser={ mapping.parser }"
+            )
+
     def get_default_mapping(self, mappings: (list, None) = None):
         """
         Gets the mapping with the `default` pattern from a list of mappings.
@@ -511,7 +522,7 @@ def parser(self):
     def parser(self, value):
         """Sets the parser attribute after validating the new value."""
         # Require parser to have a base class of mail2beyond.framework.BaseParser
-        if inspect.isclass(value) and hasattr(value, "parse_content"):
+        if inspect.isclass(value) and issubclass(value, BaseParser):
             self._parser = value
         else:
             raise TypeError("parser must be a class (not object) with base class 'BaseParser'")

mail2beyond/tools.py

@@ -10,6 +10,7 @@
 
 from OpenSSL import crypto
 
+import mail2beyond.framework
 from . import connectors
 from . import parsers
 from . import framework
@@ -54,14 +55,23 @@ def get_connector_modules(path: (str, None) = None):
             try:
                 module = __import__(module_path.stem)
                 getattr(module, "Connector")
-                available_connectors[module_path.stem] = module
             except ModuleNotFoundError as exc:
                 mod_not_found_err_msg = f"failed to import connector module '{module_path.stem}' from '{path}'"
                 raise framework.Error(mod_not_found_err_msg) from exc
             except AttributeError as exc:
                 attr_err_msg = f"connector module '{module_path.stem}' from '{path}' has no class named 'Connector'"
                 raise framework.Error(attr_err_msg) from exc
 
+            # Ensure the module's Connector class is a subclass of BaseConnector
+            if inspect.isclass(module.Connector) and issubclass(module.Connector, mail2beyond.framework.BaseConnector):
+                available_connectors[module_path.stem] = module
+                continue
+
+            # Throw an error if the module's Connector class is not a subclass of BaseConnector
+            raise framework.Error(
+                f"'Connector' class in '{ module_path }' is not subclass of 'mail2beyond.framework.BaseConnector'"
+            )
+
     # Return the gathered connector modules
     return available_connectors
 
@@ -202,6 +212,16 @@ def get_parser_modules(path: (str, None) = None):
                 attr_err_msg = f"parser module '{module_path.stem}' from '{path}' has no class named 'Parser'"
                 raise framework.Error(attr_err_msg) from exc
 
+            # Ensure the module's Parser class is a subclass of BaseParser
+            if inspect.isclass(module.Parser) and issubclass(module.Parser, mail2beyond.framework.BaseParser):
+                available_parsers[module_path.stem] = module
+                continue
+
+            # Throw an error if the module's Parser class is not a subclass of BaseParser
+            raise framework.Error(
+                f"'Parser' class in '{module_path}' is not subclass of 'mail2beyond.framework.BaseParser'"
+            )
+
     # Return the gathered parser modules
     return available_parsers
 

scripts/mail2beyond

@@ -47,7 +47,9 @@ class Mail2BeyondCLI:
         self.args = self.parse_args()
         self.listeners = mail2beyond.tools.get_listeners_from_dict(
             config=self.args.config,
-            log_level=logging.DEBUG if self.args.verbose else logging.INFO
+            log_level=logging.DEBUG if self.args.verbose else logging.INFO,
+            connectors_path=self.args.connector_plugins_dir,
+            parsers_path=self.args.parser_plugins_dir
         )
 
         # Start the listeners
@@ -70,6 +72,20 @@ class Mail2BeyondCLI:
             required=True,
             help="Set the path to the config file"
         )
+        parser.add_argument(
+            '--connector-plugins-dir',
+            dest="connector_plugins_dir",
+            required=False,
+            default=None,
+            help="Set the path to a directory that contains plugin connector modules"
+        )
+        parser.add_argument(
+            '--parser-plugins-dir',
+            dest="parser_plugins_dir",
+            required=False,
+            default=None,
+            help="Set the path to a directory that contains plugin parser modules"
+        )
         parser.add_argument(
             '--version', "-V",
             action='version',