lib, bots, doc: STOMP/*n6*-related fixes/enhancements, also ad security

SSL-related changes -- regarding `intelmq.lib.mixins.StompMixin` and,
therefore, also the *STOMP collector* bot (`StompCollectorBot` defined
in `intelmq.bots.collectors.stomp.collector`) and the *STOMP output*
bot (`StompOutputBot` defined in `intelmq.bots.outputs.stomp.output`)
-- have been made:

* *Security*-focused: fixed certain security problems which were caused
  by the fact that numerous versions of the `stomp.py` library we need
  to be compatible with use the `ssl` module's tools in such ways that
  suffer from certain *security weaknesses*. In particular, `stomp.py`
  in versions `< 4.1.12` uses a deprecated helper: `ssl.wrap_socket()`;
  and, on the other hand, `stomp.py >=8.0, <8.1` mistakenly creates an
  `SSLContext` instance with the `check_hostname` flag unset; in both
  cases, an important negative effect is that the hostname of the STOMP
  server is *not* checked during the TLS handshake (making all STOMP
  communication vulnerable to certain kinds of attacks...).

* *Admin convenience*-focused: from now on, for each of the STOMP bots,
  you can set the `ssl_ca_certificate` config param to an empty string
  -- dictating that the SSL tools employed by the `stomp.py`'s machinery
  will attempt to load the system’s default CA certificates. Thanks to
  that, administrators of the given IntelMQ instance can be relieved of
  of the fuss with manual updates of the CA certificate(s) file -- *if*
  the certificate of the STOMP server can be verified using some of
  the publicly available CA certificates which are part of nearly all
  mainstream operating system distributions (this will be the case with
  the server certificate of the new variant of the *n6* Stream API, that
  is, the variant with STOMP-login-and-passcode-based authentication).

An important part of the implementation of the aforementioned changes is
a non-public class, `intelmq.lib.mixins.stomp._StompPyDedicatedSSLProxy`
-- which implements a kind of transparent proxy object that wraps the
`ssl` attribute of the `stomp.transport` module (originaly set to the
`ssl` module object), replacing some of the `ssl` module's tools with
their patched variants (note that the `ssl` module itself and all its
members are left untouched).

The parts of the IntelMQ's documentation related to those STOMP bots +
integration with *n6* (including the CERT.PL's "N6 Stomp Stream" feed
description) have been appropriately updated and slightly improved;
also, the changelog has been updated.

CHANGELOG.md

@@ -19,26 +19,34 @@ CHANGELOG
     if `auth_by_ssl_client_certificate` is *false*);
   - `password` (STOMP authentication passcode, default: "guest"; to be used only
     if `auth_by_ssl_client_certificate` is *false*).
+- Add the possibility to set the `ssl_ca_certificate` configuration parameter for
+  `intelmq.bots.collectors.stomp.collector` and/or `intelmq.bots.outputs.stomp.output`
+  to an empty string - which means that the SSL machinery used for STOMP communication
+  will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).
 
 ### Core
 - `intelmq.lib.message`: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
 - `intelmq.lib.exceptions.InvalidKey`: Add optional parameter `additional_text` (PR#2398 by Sebastian Wagner).
 - `intelmq.lib.mixins`: Add a new class, `StompMixin` (defined in a new submodule: `stomp`),
   which provides certain common STOMP-bot-specific operations, factored out from
   `intelmq.bots.collectors.stomp.collector` and `intelmq.bots.outputs.stomp.output`
-  (PR#2408 by Jan Kaliszewski).
+  (PR#2408 and PR#2414 by Jan Kaliszewski).
 
 ### Development
 
 ### Data Format
 
 ### Bots
 #### Collectors
-- `intelmq.bots.collectors.stomp.collector` (PR#2408 by Jan Kaliszewski):
+- `intelmq.bots.collectors.stomp.collector` (PR#2408 and PR#2414 by Jan Kaliszewski):
   - Add support for authentication based on STOMP login and passcode, introducing three
     new configuration parameters (see above: *Configuration*).
+  - Add support for loading the system’s default CA certificates, as an alternative to
+    specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
   - Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`);
     fixes [#2342](https://github.com/certtools/intelmq/issues/2342).
+  - Fix (by carefully targeted monkey patching) certain security problems caused by
+    SSL-related weaknesses that various versions of `stomp.py` suffer from.
   - Fix the reconnection behavior: do not attempt to reconnect after `shutdown`. Also,
     never attempt to reconnect if the version of `stomp.py` is older than `4.1.21` (it
     did not work properly anyway).
@@ -54,10 +62,14 @@ CHANGELOG
 #### Experts
 
 #### Outputs
-- `intelmq.bots.outputs.stomp.output` (PR#2408 by Jan Kaliszewski):
+- `intelmq.bots.outputs.stomp.output` (PR#2408 and PR#2414 by Jan Kaliszewski):
   - Add support for authentication based on STOMP login and passcode, introducing three
     new configuration parameters (see above: *Configuration*).
+  - Add support for loading the system’s default CA certificates, as an alternative to
+    specifying the CA certificate(s) file path explicitly (see above: *Configuration*).
   - Update the code to support new versions of `stomp.py`, including the latest (`8.1.0`).
+  - Fix (by carefully targeted monkey patching) certain security problems caused by
+    SSL-related weaknesses that various versions of `stomp.py` suffer from.
   - Fix `AttributeError` caused by attempts to get unset attributes of `StompOutputBot`
     (`ssl_ca_cert` et consortes).
   - Add coercion of the `port` config parameter to `int`.
@@ -69,7 +81,7 @@ CHANGELOG
 ### Documentation
 - Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
 - Update/fix/improve the stuff related to the STOMP bots and integration with the *n6*'s
-  Stream API (PR#2408 by Jan Kaliszewski).
+  Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
 
 ### Packaging
 

docs/user/bots.rst

@@ -949,7 +949,7 @@ Install the `stomp.py` library from PyPI:
 * `port`: STOMP server's port number (default: 61614)
 * `exchange`: STOMP *destination* to subscribe to, e.g. "/exchange/my.example.org/*.*.*.*"
 * `heartbeat`: default: 6000
-* `ssl_ca_certificate`: path to CA file
+* `ssl_ca_certificate`: path to CA file, or empty string to load system's default CA certificates
 * `auth_by_ssl_client_certificate`: Boolean, default: true (note: false is needed for new *n6* auth)
 * `ssl_client_certificate`: path to client cert file, used only if `auth_by_ssl_client_certificate` is true
 * `ssl_client_certificate_key`: path to client cert key file, used only if `auth_by_ssl_client_certificate` is true
@@ -4313,7 +4313,7 @@ Also you will need a so called "exchange point".
 * `port`: STOMP server's port number (default: 61614)
 * `exchange`: STOMP *destination* to push at, e.g. "/exchange/_push" (which is default)
 * `heartbeat`: default: 60000
-* `ssl_ca_certificate`: path to CA file
+* `ssl_ca_certificate`: path to CA file, or empty string to load system's default CA certificates
 * `auth_by_ssl_client_certificate`: Boolean, default: true (note: false is needed for new *n6* auth)
 * `ssl_client_certificate`: path to client cert file, used only if `auth_by_ssl_client_certificate` is true
 * `ssl_client_certificate_key`: path to client cert key file, used only if `auth_by_ssl_client_certificate` is true

intelmq/bots/collectors/stomp/collector.py

@@ -83,9 +83,14 @@ class StompCollectorBot(CollectorBot, StompMixin):
     exchange: str = ''
     heartbeat: int = 6000
 
-    # Note: the `ssl_ca_certificate` configuration parameter must always
-    # be set to the server's CA certificate(s) file path.
-    ssl_ca_certificate: str = 'ca.pem'
+    # Note: the `ssl_ca_certificate` configuration parameter must be set:
+    # * *either* to the server's CA certificate(s) file path,
+    # * *or* to an empty string -- dictating that the SSL tools employed
+    #   by the `stomp.py`'s machinery will attempt to load the system’s
+    #   default CA certificates.
+    # The latter, if applicable, is more convenient -- by avoiding the
+    # need to manually update the CA certificate(s) file.
+    ssl_ca_certificate: str = 'ca.pem'  # <- TODO: change to '' (+ remove "ca.pem*" legacy files)
     # (^ TODO: could also be pathlib.Path)
 
     auth_by_ssl_client_certificate: bool = True

intelmq/bots/outputs/stomp/output.py

@@ -24,14 +24,19 @@ class StompOutputBot(OutputBot, StompMixin):
     message_with_type: bool = False
     single_key: bool = False
 
-    server: str = '127.0.0.1'  # TODO: could be ip address
+    server: str = '127.0.0.1'  # <- TODO: change to 'n6stream.cert.pl' (==StompCollectorBot.server)
     port: int = 61614
     exchange: str = '/exchange/_push'
     heartbeat: int = 60000
 
-    # Note: the `ssl_ca_certificate` configuration parameter must always
-    # be set to the server's CA certificate(s) file path.
-    ssl_ca_certificate: str = 'ca.pem'
+    # Note: the `ssl_ca_certificate` configuration parameter must be set:
+    # * *either* to the server's CA certificate(s) file path,
+    # * *or* to an empty string -- dictating that the SSL tools employed
+    #   by the `stomp.py`'s machinery will attempt to load the system’s
+    #   default CA certificates.
+    # The latter, if applicable, is more convenient -- by avoiding the
+    # need to manually update the CA certificate(s) file.
+    ssl_ca_certificate: str = 'ca.pem'  # <- TODO: change to ''
     # (^ TODO: could also be pathlib.Path)
 
     auth_by_ssl_client_certificate: bool = True

intelmq/etc/feeds.yaml

@@ -1153,14 +1153,36 @@ providers:
         Note that 'rate_limit' does not apply to this bot, as it is waiting for messages
         on a stream.
       additional_information: Contact CERT.pl to get access to the feed.
+        Note that the configuration parameter values suggested here are
+        proper to the new *n6* Stream API variant (with authentication
+        based on 'username' and 'password'); for this variant, typically
+        you can leave the 'ssl_ca_certificate' parameter's value empty -
+        then the system's default CA certificates will be used; however,
+        if that does not work, you need to set 'ssl_ca_certificate' to
+        the path to a file containing CA certificates eligible to verify
+        "*.cert.pl" server certificates (to be found among the publicly
+        available CA certs distributed with modern web browsers/OSes).
+        Also, note that the 'server' parameter's value (for the *new API
+        variant*) suggested here, "n6stream-new.cert.pl", is a temporary
+        domain; ultimately, it will be changed back to "stream.cert.pl".
+        When it comes to the *old API variant* (turned off in November
+        2023!), you need to have the 'server' parameter set to the name
+        "n6stream.cert.pl", 'auth_by_ssl_client_certificate' set to
+        true, 'ssl_ca_certificate' set to the path to a file containing
+        the *n6*'s legacy self-signed CA certificate (which is stored in
+        file "intelmq/bots/collectors/stomp/ca.pem"), and the parameters
+        'ssl_client_certificate' and 'ssl_client_certificate_key' set to
+        the paths to your-*n6*-client-specific certificate and key files
+        (note that the 'username' and 'password' parameters are then
+        irrelevant and can be omitted).
       bots:
         collector:
           module: intelmq.bots.collectors.stomp.collector
           parameters:
             exchange: "{insert your STOMP *destination* to subscribe to, as given by CERT.pl, e.g. /exchange/my.example.org/*.*.*.*}"
-            server: n6stream.cert.pl
+            server: "n6stream-new.cert.pl"
             port: 61614
-            ssl_ca_certificate: "{insert path to CA file for CERT.pl's n6}"
+            ssl_ca_certificate: ""
             auth_by_ssl_client_certificate: false
             username: "{insert your *n6* login, e.g. [email protected]}"
             password: "{insert your *n6* API key}"
@@ -1169,7 +1191,7 @@ providers:
         parser:
           module: intelmq.bots.parsers.n6.parser_n6stomp
           parameters:
-      revision: 2023-09-23
+      revision: 2023-10-08
       documentation: https://n6.readthedocs.io/usage/streamapi/
       public: false
   AlienVault: