[Sentinel] Split former Sentinel connector into two new connectors (external import and stream)

external-import/sentinel-incidents/.dockerignore

@@ -0,0 +1,5 @@
+src/config.yml
+src/__pycache__
+src/logs
+src/*.gql
+src/.venv

external-import/sentinel-incidents/Dockerfile

@@ -0,0 +1,19 @@
+FROM python:3.11-alpine
+ENV CONNECTOR_TYPE=EXTERNAL_IMPORT
+
+# Copy the connector
+COPY src /opt/opencti-sentinel-incidents
+
+# Install Python modules
+# hadolint ignore=DL3003
+RUN apk update && apk upgrade && \
+    apk --no-cache add git build-base libmagic libffi-dev libxml2-dev libxslt-dev
+
+RUN cd /opt/opencti-sentinel-incidents && \
+    pip3 install --no-cache-dir -r requirements.txt && \
+    apk del git build-base
+
+# Expose and entrypoint
+COPY entrypoint.sh /
+RUN chmod +x /entrypoint.sh
+ENTRYPOINT ["/entrypoint.sh"]

external-import/sentinel-incidents/README.md

@@ -0,0 +1,184 @@
+# OpenCTI Sentinel Incidents Connector
+
+<!--
+General description of the connector
+* What it does
+* How it works
+* Special requirements
+* Use case description
+* ...
+-->
+
+Table of Contents
+
+- [OpenCTI Sentinel Incidents Connector](#opencti-sentinel-incidents-connector)
+    - [Introduction](#introduction)
+    - [Installation](#installation)
+        - [Requirements](#requirements)
+    - [Configuration variables](#configuration-variables)
+        - [OpenCTI environment variables](#opencti-environment-variables)
+        - [Base connector environment variables](#base-connector-environment-variables)
+        - [Connector extra parameters environment variables](#connector-extra-parameters-environment-variables)
+    - [Deployment](#deployment)
+        - [Docker Deployment](#docker-deployment)
+        - [Manual Deployment](#manual-deployment)
+    - [Usage](#usage)
+    - [Behavior](#behavior)
+    - [Debugging](#debugging)
+    - [Additional information](#additional-information)
+
+## Introduction
+
+This OpenCTI connector allows to import indicators and observables from Microsoft Sentinel's incidents to your OpenCTI
+platform.
+
+## Installation
+
+If you don't know how to get the `tenant_id`, `client_id`, and `client_secret` information, here's a screenshot to
+help !
+![Sentinel_variables](./doc/sentinel_info_variables.png)
+
+It's also important to define the necessary permissions in Sentinel for the connector to work.
+
+In the Azure portal, you need to set :
+Home > Application Registration > OpenCTI (your name) > API Permissions
+and prioritize the "ThreatIndicators.ReadWrite.OwnedBy" permissions.
+![Sentinel_permission](./doc/permission_mandatory.png)
+You will then be able to view the data (indicators) in :
+Home > Microsoft Sentinel > OpenCTI (Your Name) > Threat Indicators
+
+For more information, visit:
+
+- [Microsoft Security-Authorization](https://learn.microsoft.com/en-us/graph/security-authorization)
+- [Microsoft Connect-Threat-Intelligence-Tip](https://learn.microsoft.com/en-us/azure/sentinel/connect-threat-intelligence-tip)
+
+Another interesting link:
+
+- [Microsoft Sentinel-Threat-Intelligence](https://learn.microsoft.com/en-us/azure/architecture/example-scenario/data/sentinel-threat-intelligence#import-threat-indicators-with-the-platforms-data-connector)
+
+### Requirements
+
+- OpenCTI Platform >= 6.4
+
+## Configuration variables
+
+There are a number of configuration options, which are set either in `docker-compose.yml` (for Docker) or
+in `config.yml` (for manual deployment).
+
+### OpenCTI environment variables
+
+Below are the parameters you'll need to set for OpenCTI:
+
+| Parameter     | config.yml `opencti` | Docker environment variable | Mandatory | Description                                          |
+|---------------|----------------------|-----------------------------|-----------|------------------------------------------------------|
+| OpenCTI URL   | `url`                | `OPENCTI_URL`               | Yes       | The URL of the OpenCTI platform.                     |
+| OpenCTI Token | `token`              | `OPENCTI_TOKEN`             | Yes       | The default admin token set in the OpenCTI platform. |
+
+### Base connector environment variables
+
+Below are the parameters you'll need to set for running the connector properly:
+
+| Parameter       | config.yml `connector` | Docker environment variable | Default         | Mandatory | Description                                                                              |
+|-----------------|------------------------|-----------------------------|-----------------|-----------|------------------------------------------------------------------------------------------|
+| Connector ID    | `id`                   | `CONNECTOR_ID`              | /               | Yes       | A unique `UUIDv4` identifier for this connector instance.                                |
+| Connector Type  | `type`                 | `CONNECTOR_TYPE`            | EXTERNAL_IMPORT | Yes       | Should always be set to `EXTERNAL_IMPORT` for this connector.                            |
+| Connector Name  | `name`                 | `CONNECTOR_NAME`            |                 | Yes       | Name of the connector.                                                                   |
+| Connector Scope | `scope`                | `CONNECTOR_SCOPE`           |                 | Yes       | The scope or type of data the connector is importing, either a MIME type or Stix Object. |
+| Log Level       | `log_level`            | `CONNECTOR_LOG_LEVEL`       | info            | Yes       | Determines the verbosity of the logs. Options are `debug`, `info`, `warn`, or `error`.   |
+
+### Connector extra parameters environment variables
+
+Below are the parameters you'll need to set for the connector:
+
+| Parameter         | config.yml `sentinel_incidents` | Docker environment variable            | Default | Mandatory | Description                                                                             |
+|-------------------|---------------------------------|----------------------------------------|---------|-----------|-----------------------------------------------------------------------------------------|
+| Tenant ID         | `tenant_id`                     | `SENTINEL_INCIDENTS_TENANT_ID`         |         | Yes       | Your Azure App Tenant ID, see the screenshot to help you find this information.         |
+| Client ID         | `client_id`                     | `SENTINEL_INCIDENTS_CLIENT_ID`         |         | Yes       | Your Azure App Client ID, see the screenshot to help you find this information.         |
+| Client Secret     | `client_secret`                 | `SENTINEL_INCIDENTS_CLIENT_SECRET`     |         | Yes       | Your Azure App Client secret, See the screenshot to help you find this information.     |
+| Login URL         | `login_url`                     | `SENTINEL_INCIDENTS_LOGIN_URL`         |         | Yes       | Login URL for Microsoft which is `https://login.microsoft.com`                          |
+| API Base URL      | `api_base_url`                  | `SENTINEL_INCIDENTS_API_BASE_URL`      |         | Yes       | The resource the API will use which is `https://graph.microsoft.com`                    |
+| Incident URL Path | `incident_path`                 | `SENTINEL_INCIDENTS_INCIDENT_PATH`     |         | Yes       | The incident URL that will be used which is `/v1.0/security/incidents`                  |
+| Target Product    | `target_product`                | `SENTINEL_INCIDENTS_TARGET_PRODUCT`    |         | Yes       | `Azure Sentinel` or `Microsoft Defender` ATP                                            |
+| Import start date | `import_start_date`             | `SENTINEL_INCIDENTS_IMPORT_START_DATE` |         | Yes       | Import starting date (in YYYY-MM-DD format) - used only if connector's state is not set |
+
+## Deployment
+
+### Docker Deployment
+
+Before building the Docker container, you need to set the version of pycti in `requirements.txt` equal to whatever
+version of OpenCTI you're running. Example, `pycti==6.3.4`. If you don't, it will take the latest version, but
+sometimes the OpenCTI SDK fails to initialize.
+
+Build a Docker Image using the provided `Dockerfile`.
+
+Example:
+
+```shell
+# Replace the IMAGE NAME with the appropriate value
+docker build . -t [IMAGE NAME]:latest
+```
+
+Make sure to replace the environment variables in `docker-compose.yml` with the appropriate configurations for your
+environment. Then, start the docker container with the provided docker-compose.yml
+
+```shell
+docker compose up -d
+# -d for detached
+```
+
+### Manual Deployment
+
+Create a file `config.yml` based on the provided `config.yml.sample`.
+
+Replace the configuration variables (especially the "**ChangeMe**" variables) with the appropriate configurations for
+you environment.
+
+Install the required python dependencies (preferably in a virtual environment):
+
+```shell
+pip3 install -r requirements.txt
+```
+
+Then, start the connector from recorded-future/src:
+
+```shell
+python3 main.py
+```
+
+## Usage
+
+After Installation, the connector should require minimal interaction to use, and should update automatically at a
+regular interval specified in your `docker-compose.yml` or `config.yml` in `duration_period`.
+
+However, if you would like to force an immediate download of a new batch of entities, navigate to:
+
+`Data management` -> `Ingestion` -> `Connectors` in the OpenCTI platform.
+
+Find the connector, and click on the refresh button to reset the connector's state and force a new
+download of data by re-running the connector.
+
+## Behavior
+
+<!--
+Describe how the connector functions:
+* What data is ingested, updated, or modified
+* Important considerations for users when utilizing this connector
+* Additional relevant details
+-->
+
+## Debugging
+
+The connector can be debugged by setting the appropiate log level.
+Note that logging messages can be added using `self.helper.connector_logger,{LOG_LEVEL}("Sample message")`, i.
+e., `self.helper.connector_logger.error("An error message")`.
+
+<!-- Any additional information to help future users debug and report detailed issues concerning this connector -->
+
+## Additional information
+
+<!--
+Any additional information about this connector
+* What information is ingested/updated/changed
+* What should the user take into account when using this connector
+* ...
+-->

external-import/sentinel-incidents/docker-compose.yml

@@ -0,0 +1,44 @@
+version: '3'
+services:
+  connector-sentinel-incidents:
+    image: opencti/connector-sentinel-incidents:6.2.4
+    environment:
+      # Connector's generic execution parameters
+      - OPENCTI_URL=http://localhost
+      - OPENCTI_TOKEN=CHANGEME
+
+      # Connector's definition parameters REQUIRED
+      - CONNECTOR_ID=CHANGEME
+      - "CONNECTOR_NAME=Microsoft Sentinel"
+      - CONNECTOR_SCOPE=sentinel # MIME type or Stix Object - Not used
+      - CONNECTOR_LOG_LEVEL=error
+      # Connector's definition parameters OPTIONAL
+      # - CONNECTOR_QUEUE_THRESHOLD=500 # Default 500Mo, Float accepted
+      # - CONNECTOR_RUN_AND_TERMINATE=False # Default False, True run connector once
+      # - CONNECTOR_SEND_TO_QUEUE=True # Default True
+      # - CONNECTOR_SEND_TO_DIRECTORY=False # Default False 
+      # - CONNECTOR_SEND_TO_DIRECTORY_PATH=CHANGEME # if CONNECTOR_SEND_TO_DIRECTORY is True, you must specify a path
+      # - CONNECTOR_SEND_TO_DIRECTORY_RETENTION=7 # Default 7, in days
+
+      # Connector's custom execution parameters
+      - SENTINEL_INCIDENTS_TENANT_ID=ChangeMe # Azure Tentant ID
+      - SENTINEL_INCIDENTS_WORKSPACE_ID=ChangeMe # Sentinel Workspace ID (only for Azure Sentinel)
+      - SENTINEL_INCIDENTS_CLIENT_ID=ChangeMe # Azure App Client ID
+      - SENTINEL_INCIDENTS_CLIENT_SECRET=ChangeMe # Azure App Client Secret
+      - SENTINEL_INCIDENTS_"TARGET_PRODUCT=Azure Sentinel" # "Azure Sentinel" or "Microsoft Defender ATP"
+      - SENTINEL_INCIDENTS_LOGIN_URL=https://login.microsoft.com
+      - SENTINEL_INCIDENTS_API_BASE_URL=https://graph.microsoft.com
+      - SENTINEL_INCIDENTS_INCIDENT_PATH=/v1.0/security/incidents
+
+      # Add proxy parameters below if needed
+      # - HTTP_PROXY=CHANGEME
+      # - HTTPS_PROXY=CHANGEME
+      # - NO_PROXY=CHANGEME
+    restart: always
+    # networks:
+    #   - docker_default
+
+# networks:
+#   default:
+#     external: true
+#     name: docker_default

external-import/sentinel-incidents/entrypoint.sh

@@ -0,0 +1,7 @@
+#!/bin/sh
+
+# Go to the right directory
+cd /opt/opencti-connector-sentinel-incidents
+
+# Launch the worker
+python3 main.py

external-import/sentinel-incidents/src/config.yml.sample

@@ -0,0 +1,32 @@
+opencti:
+  url: 'http://localhost:PORT'
+  token: 'ChangeMe'
+
+connector:
+  id: 'ChangeMe'
+  type: 'EXTERNAL_IMPORT'
+  name: 'External Import Connector Template'
+  scope: 'ChangeMe'
+  log_level: 'info'
+  duration_period: 'PT5M' # Interval given for scheduler process in ISO-8601 format
+  #============================================#
+  # Optional connector's definition parameters #
+  #============================================#
+  #queue_threshold: 500
+  #run_and_terminate: 'False'
+  #send_to_queue: 'True'
+  #send_to_directory: 'False'
+  #send_to_directory_path: 'ChangeMe'
+  #send_to_directory_retention: 7
+
+sentinel_incidents:
+  tenant_id: 'ChangeMe'
+  workspace_id: 'ChangeMe' # Only for Sentinel
+  client_id: 'ChangeMe'
+  client_secret: 'ChangeMe'
+  login_url: 'https://login.microsoft.com'
+  api_base_url: 'https://graph.microsoft.com'
+  incident_path : '/v1.0/security/incidents'
+  confidence_level: 50
+  target_product: 'Azure Sentinel'
+  import_start_date: ''

external-import/sentinel-incidents/src/main.py

@@ -0,0 +1,20 @@
+import traceback
+
+from sentinel_incidents_connector import SentinelIncidentsConnector
+
+if __name__ == "__main__":
+    """
+    Entry point of the script
+
+    - traceback.print_exc(): This function prints the traceback of the exception to the standard error (stderr).
+    The traceback includes information about the point in the program where the exception occurred,
+    which is very useful for debugging purposes.
+    - exit(1): effective way to terminate a Python program when an error is encountered.
+    It signals to the operating system and any calling processes that the program did not complete successfully.
+    """
+    try:
+        connector = SentinelIncidentsConnector()
+        connector.run()
+    except Exception:
+        traceback.print_exc()
+        exit(1)

external-import/sentinel-incidents/src/requirements.txt

@@ -0,0 +1 @@
+pycti==6.3.1

external-import/sentinel-incidents/src/sentinel_incidents_connector/__init__.py

@@ -0,0 +1,3 @@
+from .connector import SentinelIncidentsConnector
+
+__all__ = ["SentinelIncidentsConnector"]

external-import/sentinel-incidents/src/sentinel_incidents_connector/client_api.py

@@ -0,0 +1,68 @@
+import json
+
+import requests
+
+
+class ConnectorClient:
+    def __init__(self, helper, config):
+        """
+        Initialize the client with necessary configurations
+        """
+        self.helper = helper
+        self.config = config
+        self.headers = None
+
+        self.generate_oauth_token()
+
+    def _request_data(self, api_url: str, params=None):
+        """
+        Internal method to handle API requests
+        :return: Response in JSON format
+        """
+        try:
+            response = self.session.get(api_url, params=params)
+
+            self.helper.connector_logger.info(
+                "[API] HTTP Get Request to endpoint", {"url_path": api_url}
+            )
+
+            response.raise_for_status()
+            return response
+
+        except requests.RequestException as err:
+            error_msg = "[API] Error while fetching data: "
+            self.helper.connector_logger.error(
+                error_msg, {"url_path": {api_url}, "error": {str(err)}}
+            )
+            return None
+
+    def generate_oauth_token(self):
+        try:
+            url = f"https://login.microsoftonline.com/{self.config.tenant_id}/oauth2/v2.0/token"
+            oauth_data = {
+                "client_id": self.config.client_id,
+                "client_secret": self.config.client_secret,
+                "grant_type": "client_credentials",
+                "scope": "https://graph.microsoft.com/.default",
+            }
+            response = requests.post(url, data=oauth_data)
+            response_json = json.loads(response.text)
+
+            oauth_token = response_json["access_token"]
+            self.headers = {"Authorization": oauth_token}
+        except Exception as e:
+            raise ValueError("[ERROR] Failed generating oauth token {" + str(e) + "}")
+
+    def get_incidents(self) -> list[dict]:
+        """
+        Get incidents with their alerts from Microsft Sentinel API.
+        :return: List of incidents
+        """
+        try:
+            url = (
+                f"{self.config.api_base_url}{self.config.incident_path}?$expand=alerts"
+            )
+            response = requests.get(url, headers=self.headers)
+            return response.json()["value"] if "value" in response.json() else []
+        except Exception as err:
+            self.helper.connector_logger.error(err)