Merge pull request #18 from soluna-studios/cli_enhancements

MANIFEST.in

@@ -1,2 +1,3 @@
 include README.md
 include requirements.txt
+include LICENSE

docs/DOCUMENTATION.md

@@ -484,12 +484,14 @@ configuration file.
 
 ## Arguments
 
-| Argument               | Required | Description                                                        |
-|------------------------|:---------|--------------------------------------------------------------------|
-| `--config (-c) <FILE>` | Yes      | Specifies a config file to load. This must be a JSON or YAML file. |
-| `--version (-V)`       | No       | Prints the version of mail2beyond installed.                       |
-| `--verbose (-v)`       | No       | Enables verbose logging.                                           |
-| `--help (-h)`          | No       | Prints the help page.                                              |
+| Argument                 | Required | Description                                                          |
+|--------------------------|:---------|----------------------------------------------------------------------|
+| `--config (-c) <FILE>`   | Yes      | Specifies a config file to load. This must be a JSON or YAML file.   |
+| `--connector-plugin-dir` | No       | Specifies a path to a directory containing plugin connector modules. |
+| `--parser-plugin-dir`    | No       | Specifies a path to a directory containing plugin parser modules.    |
+| `--version (-V)`         | No       | Prints the version of mail2beyond installed.                         |
+| `--verbose (-v)`         | No       | Enables verbose logging.                                             |
+| `--help (-h)`            | No       | Prints the help page.                                                |
 
 ## Configuration
 A configuration file must be written before ```mail2beyond``` is started. It may also be helpful to check out the 
@@ -572,7 +574,8 @@ configuration. This name will be used to assign this connector to mappings in yo
 
 - _Required_: Yes
 - _Options_: [`void`, `smtp`, `slack`, `discord`, `google_chat`, `microsoft_teams`]
-- _Description_: The module this connector will use. Multiple connectors can use the same underlying module.
+- _Description_: The module this connector will use. Multiple connectors can use the same underlying module. You can 
+also [use your own custom connector modules](#using-plugin-connector-and-parser-modules).
 
 **config**
 
@@ -605,10 +608,9 @@ configuration. Multiple mappings can use the same the connector.
 - _Required_: No
 - _Options_: [`auto`, `plain`, `html`]
 - _Default_: `auto`
-- _Description_: Explicitly set the content-type parser to use. By default, the `auto` parser will be chosen to 
-select the parser that best matches the SMTP message's content-type header. The `plain` parser will not parse the
-content body and is the fallback for the `auto` parser if no parser exists for the content-type. The `html` parser
-will parse the content body as HTML and convert it to a more human-readable markdown format.
+- _Description_: Explicitly set the content-type parser to use. See the [built-in parsers section](#built-in-parsers)
+for more information on available parsers. You can also 
+[use your own custom parser modules](#using-plugin-connector-and-parser-modules).
 - _Notes_:
   - Some connector modules do not respect the `parser` option (e.g. `smtp`, `void`)
   - Even though the `html` parser converts the content to markdown, this does not guarantee the markdown content will
@@ -627,6 +629,63 @@ Once you have your configuration file written, you can start the server by runni
 mail2beyond --config /path/to/your/config.yml
 ```
 
+## Using Plugin Connector and Parser Modules
+If you have written your own [connector modules](#writing-custom-connectors) and/or 
+[parser modules](#writing-custom-parsers), you can include them from the CLI using the `--connector-plugins-dir` and 
+`--parser-plugins-dir` commands respectively. If specified, both these commands require a path to an existing directory 
+where your plugin connector and parser modules are stored. In order for plugins to be included, your modules must use 
+the `.py` extension within the specified directory. Additionally, extended `BaseConnector` classes must be named 
+`Connector` and extended `BaseParser` classes must be named `Parser` in order for them to be used via CLI. 
+
+In example, say we have a file structure like this:
+
+```
+plugins/
+├─ connectors/
+│  ├─ my_custom_connector_module_1.py
+│  ├─ my_custom_connector_module_2.py
+├─ parsers/
+│  ├─ my_custom_parser_1.py
+```
+
+In your configuration, you can utilize the custom connectors and parsers like such:
+
+```yaml
+---
+listeners:
+  - address: localhost
+    port: 25
+
+mappings:
+  - pattern: default
+    field: to
+    parser: my_custom_parser_1
+    connector: custom_connector_1
+
+  - pattern: "some custom pattern"
+    field: to
+    parser: my_custom_parser_1
+    connector: custom_connector_2
+
+connectors:
+  - name: custom_connector_1
+    module: my_custom_connector_module_1
+
+  - name: custom_connector_2
+    module: my_custom_connector_module_2
+```
+
+And finally, you can start the server and set the plugin module directories:
+```commandline
+mail2beyond --config /path/to/your/config.yml --connector-plugins-dir plugins/connectors/ --parser-plugins-dir plugins/parsers/ 
+```
+
+**Notes**
+
+- Even when you specify plugin modules, you will still have access to all the built-in connectors and parsers. In the 
+  event that your custom module uses the same name as a built-in module, _your module takes priority over the built-in
+  parser and the built-in parser will not be available to use.
+
 # Built-in Connectors
 Connector modules may contain their own configurable options and requirements. Below are the available configuration
 options available to each built-in connector module:

mail2beyond/framework.py

@@ -127,6 +127,7 @@ def start(self):
         """
         self.controller.start()
         self.log.info(f"mail2beyond started listening on {self.address}:{self.port}")
+        self.log_mappings()
 
     @staticmethod
     def wait():
@@ -204,6 +205,16 @@ def setup_logging(self, level: int = logging.NOTSET, handler=None, **kwargs):
         if level == logging.DEBUG:
             self.log.warning("logging at level DEBUG may expose sensitive information in logs")
 
+    def log_mappings(self):
+        """Logs the configured mappings upon startup for debugging."""
+        # Loop through all configured mappings and log it's configuration.
+        for mapping in self.mappings:
+            self.log.debug(
+                f"loaded mapping with pattern='{ mapping.pattern }' field='{ mapping.field }'"
+                f" connector='{ mapping.connector }' connector_module='{ mapping.connector.__module__ }'"
+                f" parser={ mapping.parser }"
+            )
+
     def get_default_mapping(self, mappings: (list, None) = None):
         """
         Gets the mapping with the `default` pattern from a list of mappings.
@@ -511,7 +522,7 @@ def parser(self):
     def parser(self, value):
         """Sets the parser attribute after validating the new value."""
         # Require parser to have a base class of mail2beyond.framework.BaseParser
-        if inspect.isclass(value) and hasattr(value, "parse_content"):
+        if inspect.isclass(value) and issubclass(value, BaseParser):
             self._parser = value
         else:
             raise TypeError("parser must be a class (not object) with base class 'BaseParser'")

mail2beyond/tools.py

@@ -6,19 +6,85 @@
 import logging
 import pathlib
 import ssl
+import sys
 
 from OpenSSL import crypto
 
+import mail2beyond.framework
 from . import connectors
 from . import parsers
 from . import framework
 
 
-def get_connectors_from_dict(config: dict):
+def get_connector_modules(path: (str, None) = None):
+    """
+    Gathers all available connector modules. This allows a 'path' to be specified to optionally pass in plugin connector
+    modules. Built-in connectors are always included.
+
+    Args:
+        path (str, None): A path to a directory that contains plugin connector modules. Only .py files within this
+            directory will be included. Each .py file must include a class named `Connector` that extends the
+            `mail2beyond.framework.BaseConnector class. If `None` is specified, only the built-in connector modules
+            will be available.
+
+    Raises:
+        mail2beyond.framework.Error: When plugin connector modules could not be loaded.
+
+    Returns:
+        dict: A dictionary of available connector modules. The dictionary keys will be the module names and the values
+            will be the module itself.
+    """
+    # Start by gathering the built-in connectors from the mail2beyond.connectors sub-package.
+    available_connectors = dict(inspect.getmembers(connectors, inspect.ismodule))
+
+    # If a plugin path was passed in, include modules within that directory as well.
+    if path:
+        # Convert 'path' into an object
+        path_obj = pathlib.Path(path)
+
+        # Require path to be an existing directory
+        if not path_obj.exists() or not path_obj.is_dir():
+            raise framework.Error(f"failed to load connector modules '{path}' is not an existing directory")
+
+        # Add this directory to our Python path
+        sys.path.append(str(path_obj.absolute()))
+
+        # Loop through each .py file in the directory and ensure it is valid
+        for module_path in path_obj.glob("*.py"):
+            # Verify this module could be imported, contains the Connector class and is added to available connectors.
+            try:
+                module = __import__(module_path.stem)
+                getattr(module, "Connector")
+            except ModuleNotFoundError as exc:
+                mod_not_found_err_msg = f"failed to import connector module '{module_path.stem}' from '{path}'"
+                raise framework.Error(mod_not_found_err_msg) from exc
+            except AttributeError as exc:
+                attr_err_msg = f"connector module '{module_path.stem}' from '{path}' has no class named 'Connector'"
+                raise framework.Error(attr_err_msg) from exc
+
+            # Ensure the module's Connector class is a subclass of BaseConnector
+            if inspect.isclass(module.Connector) and issubclass(module.Connector, mail2beyond.framework.BaseConnector):
+                available_connectors[module_path.stem] = module
+                continue
+
+            # Throw an error if the module's Connector class is not a subclass of BaseConnector
+            raise framework.Error(
+                f"'Connector' class in '{ module_path }' is not subclass of 'mail2beyond.framework.BaseConnector'"
+            )
+
+    # Return the gathered connector modules
+    return available_connectors
+
+
+def get_connectors_from_dict(config: dict, path: (str, None) = None):
     """
     Converts a dictionary representations of connectors to connector objects.
     Args:
         config (dict): A dictionary representation of different connectors to create.
+        path (str, None): A path to a directory that contains plugin connector modules. Only .py files within this
+            directory will be included. Each .py file must include a class named `Connector` that extends the
+            `mail2beyond.framework.BaseConnector` class. If `None` is specified, only the built-in connector modules
+            will be available.
 
     Raises:
         mail2beyond.framework.Error: When a validation error occurs.
@@ -28,7 +94,7 @@ def get_connectors_from_dict(config: dict):
     """
     # Create a list to store the created connector objects
     valid_connectors = []
-    available_connectors = dict(inspect.getmembers(connectors, inspect.ismodule))
+    available_connectors = get_connector_modules(path)
 
     # Require connectors config to be defined
     if "connectors" not in config.keys():
@@ -99,13 +165,82 @@ def get_connector_by_name(name, connector_objs):
     return None
 
 
-def get_mappings_from_dict(config: dict):
+def get_parser_modules(path: (str, None) = None):
+    """
+    Gathers all available parser modules. This allows a 'path' to be specified to optionally pass in plugin parser
+    modules. Built-in parsers are always included.
+
+    Args:
+        path (str, None): A path to a directory that contains plugin parser modules. Only .py files within this
+            directory will be included. Each .py file must include a class named `Parser` that extends the
+            `mail2beyond.framework.BaseParser` class. If `None` is specified, only the built-in parser modules
+            will be available.
+
+    Raises:
+        mail2beyond.framework.Error: When plugin parser modules could not be loaded.
+
+    Returns:
+        dict: A dictionary of available parser modules. The dictionary keys will be the module names and the values
+            will be the module itself.
+    """
+    # Start by gathering the built-in parsers from the mail2beyond.connectors sub-package.
+    available_parsers = dict(inspect.getmembers(parsers, inspect.ismodule))
+
+    # If a plugin path was passed in, include modules within that directory as well.
+    if path:
+        # Convert 'path' into an object
+        path_obj = pathlib.Path(path)
+
+        # Require path to be an existing directory
+        if not path_obj.exists() or not path_obj.is_dir():
+            raise framework.Error(f"failed to load parser modules '{path}' is not an existing directory")
+
+        # Add this directory to our Python path
+        sys.path.append(str(path_obj.absolute()))
+
+        # Loop through each .py file in the directory and ensure it is valid
+        for module_path in path_obj.glob("*.py"):
+            # Verify this module could be imported, contains the Parser class and is added to available connectors.
+            try:
+                module = __import__(module_path.stem)
+                getattr(module, "Parser")
+                available_parsers[module_path.stem] = module
+            except ModuleNotFoundError as exc:
+                mod_not_found_err_msg = f"failed to import parser module '{module_path.stem}' from '{path}'"
+                raise framework.Error(mod_not_found_err_msg) from exc
+            except AttributeError as exc:
+                attr_err_msg = f"parser module '{module_path.stem}' from '{path}' has no class named 'Parser'"
+                raise framework.Error(attr_err_msg) from exc
+
+            # Ensure the module's Parser class is a subclass of BaseParser
+            if inspect.isclass(module.Parser) and issubclass(module.Parser, mail2beyond.framework.BaseParser):
+                available_parsers[module_path.stem] = module
+                continue
+
+            # Throw an error if the module's Parser class is not a subclass of BaseParser
+            raise framework.Error(
+                f"'Parser' class in '{module_path}' is not subclass of 'mail2beyond.framework.BaseParser'"
+            )
+
+    # Return the gathered parser modules
+    return available_parsers
+
+
+def get_mappings_from_dict(config: dict, connectors_path: (str, None) = None, parsers_path: (str, None) = None):
     """
     Converts a dictionary representations of mappings to Mapping objects. Since Mapping objects are dependent on
     a Connector object, the dictionary must also include representations of Connector objects to use.
 
     Args:
         config (dict): A dictionary representation of different mappings to create.
+        connectors_path (str, None): A path to a directory that contains plugin connector modules. Only .py files within
+            this directory will be included. Each .py file must include a class named `Connector` that extends the
+            `mail2beyond.framework.BaseConnector` class. If `None` is specified, only the built-in connector modules
+            will be available.
+        parsers_path (str, None): A path to a directory that contains plugin parser modules. Only .py files within this
+            directory will be included. Each .py file must include a class named `Parser` that extends the
+            `mail2beyond.framework.BaseParser` class. If `None` is specified, only the built-in parser modules
+            will be available.
 
     Raises:
         mail2beyond.framework.Error: When a validation error occurs.
@@ -114,8 +249,8 @@ def get_mappings_from_dict(config: dict):
         list: A list of Mapping objects that can be used.
     """
     # Variables
-    config_connectors = get_connectors_from_dict(config)
-    available_parsers = dict(inspect.getmembers(parsers, inspect.ismodule))
+    config_connectors = get_connectors_from_dict(config, path=connectors_path)
+    available_parsers = get_parser_modules(path=parsers_path)
     valid_mappings = []
 
     # Require mappings config to be defined
@@ -166,7 +301,7 @@ def get_mappings_from_dict(config: dict):
     return valid_mappings
 
 
-def get_listeners_from_dict(config: dict, log_level: int = logging.NOTSET):
+def get_listeners_from_dict(config: dict, log_level: int = logging.NOTSET, **kwargs):
     """
     Converts a dictionary representations of listeners to Listener objects. Since Listener objects are dependent on
     a Mapping objects, and Mapping objects are dependent on Connector objects, the dictionary must also include
@@ -176,6 +311,14 @@ def get_listeners_from_dict(config: dict, log_level: int = logging.NOTSET):
         config (dict): A dictionary representation of different mappings to create.
         log_level (int): Sets the logging level the Listeners' Logger will start logging at. See
             https://docs.python.org/3/library/logging.html#logging-levels
+        **connectors_path (str, None): A path to a directory that contains plugin connector modules. Only .py files
+            within this directory will be included. Each .py file must include a class named `Connector` that extends
+            the`mail2beyond.framework.BaseConnector` class. If `None` is specified, only the built-in connector modules
+            will be available.
+        **parsers_path (str, None): A path to a directory that contains plugin parser modules. Only .py files within
+            this directory will be included. Each .py file must include a class named `Parser` that extends the
+            `mail2beyond.framework.BaseParser` class. If `None` is specified, only the built-in parser modules
+            will be available.
 
     Raises:
         mail2beyond.framework.Error: When a validation error occurs.
@@ -201,8 +344,12 @@ def get_listeners_from_dict(config: dict, log_level: int = logging.NOTSET):
     tls_minimum_version_default = "tls1_2"
 
     # Variables
-    mappings = get_mappings_from_dict(config)
     valid_listeners = []
+    mappings = get_mappings_from_dict(
+        config,
+        connectors_path=kwargs.get("connectors_path"),
+        parsers_path=kwargs.get("parsers_path")
+    )
 
     # Require listeners config to be defined.
     if "listeners" not in config.keys():