Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CISA KEV] Modification on connector to use the new pycti connector helper scheduler #2396

Closed
wants to merge 4 commits into from
Closed

[CISA KEV] Modification on connector to use the new pycti connector helper scheduler #2396

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f3f69d7
[CISA KEV] Modification on connector to use the new pycti connector h…
Megafredo Jul 24, 2024
61ffdc1
[CISA KEV] Update linter
Megafredo Jul 24, 2024
d2cae45
[CISA KEV] Update linter
Megafredo Jul 24, 2024
cf98e8e
[connector] correct naming duration period
helene-nguyen Jul 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 29 additions & 17 deletions external-import/cisa-known-exploited-vulnerabilities/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,35 @@ This Connector will download the catalog from CISA's website and parse out Ident

- OpenCTI Platform >= 5.2.4

### Configuration

| Parameter | Docker envvar | Mandatory | Description |
| ------------------------------------ | ----------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `opencti_url` | `OPENCTI_URL` | Yes | The URL of the OpenCTI platform. |
| `opencti_token` | `OPENCTI_TOKEN` | Yes | The default admin token configured in the OpenCTI platform parameters file. |
| `connector_id` | `CONNECTOR_ID` | Yes | A valid arbitrary `UUIDv4` that must be unique for this connector. |
| `connector_name` | `CONNECTOR_NAME` | Yes | Option `Template` |
| `connector_scope` | `CONNECTOR_SCOPE` | Yes | Supported scope: Template Scope (MIME Type or Stix Object) |
| `connector_log_level` | `CONNECTOR_LOG_LEVEL` | Yes | The log level for this connector, could be `debug`, `info`, `warn` or `error` (less verbose). |
| `cisa_catalog_url` | `CISA_CATALOG_URL` | Yes | The URL that hosts the KEV Catalog |
| `cisa_interval` | `CISA_INTERVAL` | Yes | Interval on how often the connector should run |
| `connector_tlp` | `CONNECTOR_TLP` | Yes | TLP for data coming from this connector |

### Debugging ###

<!-- Any additional information to help future users debug and report detailed issues concerning this connector -->

### Configuration variables

Below are the parameters you'll need to set for OpenCTI:

| Parameter `OpenCTI` | config.yml | Docker environment variable | Mandatory | Description |
|---------------------|-------------|-----------------------------|-----------|------------------------------------------------------|
| URL | `url` | `OPENCTI_URL` | Yes | The URL of the OpenCTI platform. |
| Token | `token` | `OPENCTI_TOKEN` | Yes | The default admin token set in the OpenCTI platform. |

Below are the parameters you'll need to set for running the connector properly:

| Parameter `Connector` | config.yml | Docker environment variable | Default | Mandatory | Description |
|-----------------------|---------------------|-------------------------------|-------------|-----------|--------------------------------------------------------------------------------------------------|
| ID | `id` | `CONNECTOR_ID` | / | Yes | A unique `UUIDv4` identifier for this connector instance. |
| Name | `name` | `CONNECTOR_NAME` | `CISA KEV ` | Yes | Full name of the connector : `CISA Known Exploited Vulnerability`. |
| Scope | `scope` | `CONNECTOR_SCOPE` | `cisa` | Yes | Must be `cisa`, not used in this connector. |
| Run and Terminate | `run_and_terminate` | `CONNECTOR_RUN_AND_TERMINATE` | `False` | No | Launch the connector once if set to True. Takes 2 available values: `True` or `False`. |
| Duration Period | `duration_period` | `CONNECTOR_DURATION_PERIOD` | / | Yes | Determines the time interval between each launch of the connector in ISO 8601, ex: `P7D`. |
| Queue Threshold | `queue_threshold` | `CONNECTOR_QUEUE_THRESHOLD` | `500` | No | Used to determine the limit (RabbitMQ) in MB at which the connector must go into buffering mode. |
| Log Level | `log_level` | `CONNECTOR_LOG_LEVEL` | / | Yes | Determines the verbosity of the logs. Options are `debug`, `info`, `warn`, or `error`. |

Below are the parameters you'll need to set for CISA Connector:

| Parameter `Cisa` | config.yml | Docker environment variable | Default | Mandatory | Description |
|------------------------|--------------------------|-------------------------------|--------------|-----------|--------------------------------------------------------------------------------------------------------------------------|
| Catalog Url | `catalog_url` | `CISA_CATALOG_URL` | / | Yes | The URL that hosts the KEV Catalog `https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json` |
| Create Infrastructures | `create_infrastructures` | `CISA_CREATE_INFRASTRUCTURES` | / | Yes | Allows you to create or not create an infrastructure in opencti |
| TLP | `tlp` | `CISA_TLP` | `TLP:CLEAR` | Yes | TLP for data coming from this connector |

### Additional information

Expand Down
2 changes: 1 addition & 1 deletion external-import/cisa-known-exploited-vulnerabilities/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ services:
- CONNECTOR_SCOPE=cisa
- CONNECTOR_RUN_AND_TERMINATE=false
- CONNECTOR_LOG_LEVEL=error
- CONNECTOR_DURATION_PERIOD=P2D
- CISA_CATALOG_URL=https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
- CISA_CREATE_INFRASTRUCTURES=false
- CISA_TLP=TLP:CLEAR
- CISA_INTERVAL=2 # In days, must be strictly greater than 1
restart: always
4 changes: 2 additions & 2 deletions external-import/cisa-known-exploited-vulnerabilities/src/config.yml.sample
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ connector:
name: 'CISA Known Exploited Vulnerability Catalog'
scope: 'cisa'
run_and_terminate: false
duration_period: 'P2D' # Format ISO 8601
log_level: 'info'

cisa:
catalog_url: 'https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json'
create_infrastructures: false
tlp: 'TLP:CLEAR'
interval: 7 # In days, must be strictly greater than 1
tlp: 'TLP:CLEAR'
113 changes: 45 additions & 68 deletions external-import/cisa-known-exploited-vulnerabilities/src/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import ssl
import sys
import time
import traceback
import urllib
from typing import Dict, Optional

Expand All @@ -29,6 +30,20 @@ def __init__(self):
else {}
)
self.helper = OpenCTIConnectorHelper(config)

# OpenCTI config
self.opencti_url = get_config_variable(
"OPENCTI_URL", ["opencti", "url"], config
)
self.opencti_token = get_config_variable(
"OPENCTI_TOKEN", ["opencti", "token"], config
)

# Connector config
self.duration_period = get_config_variable(
"CONNECTOR_DURATION_PERIOD", ["connector", "duration_period"], config
)

# Extra config
self.cisa_catalog_url = get_config_variable(
"CISA_CATALOG_URL", ["cisa", "catalog_url"], config
Expand All @@ -39,29 +54,13 @@ def __init__(self):
config,
True,
)
self.cisa_interval = get_config_variable(
"CISA_INTERVAL", ["cisa", "interval"], config, True
)
self.tlp = get_config_variable(
"CISA_TLP", ["cisa", "tlp"], config, False, "TLP:CLEAR"
)
self.update_existing_data = get_config_variable(
"CONNECTOR_UPDATE_EXISTING_DATA",
["connector", "update_existing_data"],
config,
)

self.created_by_stix = None
self.tlp_marking = None
self.org = "Cybersecurity and Infrastructure Security Agency"
self.opencti_url = get_config_variable(
"OPENCTI_URL", ["opencti", "url"], config
)
self.opencti_token = get_config_variable(
"OPENCTI_TOKEN", ["opencti", "token"], config
)

def get_interval(self):
return int(self.cisa_interval) * 60 * 60 * 24

def retrieve_data(self, url: str) -> Optional[str]:
"""
Expand Down Expand Up @@ -155,6 +154,7 @@ def build_bundle(self, data):
created=f"{created}",
confidence=80 if description is not None and len(description) > 0 else 50,
object_marking_refs=[f"{marking_id}"],
custom_properties={"x_opencti_cisa_kev": True},
)
stix_objects.append(stix_vuln)
vuln_id = stix_vuln["id"]
Expand Down Expand Up @@ -264,47 +264,31 @@ def process_data(self):
)
)
else:
last_run = None
self.helper.log_info("Connector has never run")
# If the last_run is more than interval-1 day
if last_run is None or (
(timestamp - last_run) > ((int(self.cisa_interval) - 1) * 60 * 60 * 24)
):
self.helper.log_info("Connector will run!")

now = datetime.datetime.utcfromtimestamp(timestamp)
friendly_name = "CISA run @ " + now.strftime("%Y-%m-%d %H:%M:%S")
work_id = self.helper.api.work.initiate_work(
self.helper.connect_id, friendly_name
)
if self.cisa_catalog_url is not None and len(self.cisa_catalog_url) > 0:
cisa_data = self.retrieve_data(self.cisa_catalog_url)
self.set_created_by_stix(org=self.org)
self.set_tlp_marking(tlp_mark=self.tlp)
cisa_data = json.loads(cisa_data)
for vuln in cisa_data["vulnerabilities"]:
bundle = self.build_bundle(vuln)
self.send_bundle(work_id, bundle)
self.helper.log_info("Connector will run!")
now = datetime.datetime.utcfromtimestamp(timestamp)
friendly_name = "CISA run @ " + now.strftime("%Y-%m-%d %H:%M:%S")
work_id = self.helper.api.work.initiate_work(
self.helper.connect_id, friendly_name
)
if self.cisa_catalog_url is not None and len(self.cisa_catalog_url) > 0:
cisa_data = self.retrieve_data(self.cisa_catalog_url)
self.set_created_by_stix(org=self.org)
self.set_tlp_marking(tlp_mark=self.tlp)
cisa_data = json.loads(cisa_data)
for vuln in cisa_data["vulnerabilities"]:
bundle = self.build_bundle(vuln)
self.send_bundle(work_id, bundle)

# Store the current timestamp as a last run
message = "Connector successfully run, storing last_run as " + str(
timestamp
)
self.helper.log_info(message)
self.helper.set_state({"last_run": timestamp})
self.helper.api.work.to_processed(work_id, message)

# Store the current timestamp as a last run
message = "Connector successfully run, storing last_run as " + str(
timestamp
)
self.helper.log_info(message)
self.helper.set_state({"last_run": timestamp})
self.helper.api.work.to_processed(work_id, message)
self.helper.log_info(
"Last_run stored, next run in: "
+ str(round(self.get_interval() / 60 / 60 / 24, 2))
+ " days"
)
else:
new_interval = self.get_interval() - (timestamp - last_run)
self.helper.log_info(
"Connector will not run, next run in: "
+ str(round(new_interval / 60 / 60 / 24, 2))
+ " days"
)
except (KeyboardInterrupt, SystemExit):
self.helper.log_info("Connector stop")
sys.exit(0)
Expand All @@ -313,20 +297,14 @@ def process_data(self):

def run(self):
self.helper.log_info("Fetching CISA Known Exploited Vulnerabilities...")
get_run_and_terminate = getattr(self.helper, "get_run_and_terminate", None)
if callable(get_run_and_terminate) and self.helper.get_run_and_terminate():
self.process_data()
self.helper.force_ping()
else:
while True:
self.process_data()
time.sleep(60)
self.helper.schedule_iso(
message_callback=self.process_data, duration_period=self.duration_period
)

def send_bundle(self, work_id: str, serialized_bundle: str) -> None:
try:
self.helper.send_stix2_bundle(
serialized_bundle,
update=self.update_existing_data,
work_id=work_id,
)
except Exception as e:
Expand All @@ -337,7 +315,6 @@ def send_bundle(self, work_id: str, serialized_bundle: str) -> None:
try:
connector = Cisa()
connector.run()
except Exception as e:
print(e)
time.sleep(10)
exit(0)
except Exception:
traceback.print_exc()
sys.exit(1)