Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ACTI Indicator Query Playbook addition #18401

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
4961b62
Layout identification bug fixed
satyakidroid Mar 25, 2022
32930f7
Modified layout for alerts and reports
satyakidroid Mar 25, 2022
3ddd7af
Added playbooks
satyakidroid Mar 29, 2022
07bda45
Updated README
satyakidroid Mar 29, 2022
0f8f881
added user agent
ankitmordhwaj Mar 30, 2022
e135805
minor changes
ankitmordhwaj Mar 30, 2022
2256e8e
minor changes
ankitmordhwaj Mar 30, 2022
737fe42
minor changes
ankitmordhwaj Mar 30, 2022
3ef3ea3
Added playbooks and modified README
satyakidroid Mar 30, 2022
75f65f4
Added description to Indicator Fields and updated ACTI IntelGraph ver…
satyakidroid Mar 30, 2022
4ec2260
Update reputation-ACTI_Intelligence_Alert.json
nirmalneupane Mar 30, 2022
2087803
Update reputation-ACTI_Intelligence_Alert.json
nirmalneupane Mar 30, 2022
9fc3638
Adding description to the indicator type.
nirmalneupane Mar 30, 2022
c1db987
missing comma
nirmalneupane Mar 31, 2022
489f0ab
Update reputation-ACTI_Intelligence_Report.json
nirmalneupane Mar 31, 2022
597da20
Added description to playbooks
satyakidroid Mar 31, 2022
75954f5
description added in playbooks
ankitmordhwaj Mar 31, 2022
75fd88f
Merge branch 'ACTIIndicatorQuery_playbook' of https://github.com/iDef…
ankitmordhwaj Mar 31, 2022
f7d885f
removed unused tasks from playbook
ankitmordhwaj Mar 31, 2022
0972f22
Modified the way threat types were printed in report
satyakidroid Mar 31, 2022
84869a4
Minor changes
satyakidroid Mar 31, 2022
891aea4
Added README to playbooks
satyakidroid Mar 31, 2022
e76dd02
Added description for layouts
satyakidroid Mar 31, 2022
1b5fa5a
Changes made to incident enrichment playbook
satyakidroid Mar 31, 2022
918281b
Update README.md
satyakidroid Mar 31, 2022
0f752ec
Update README.md
satyakidroid Mar 31, 2022
35dfebc
Update README.md
satyakidroid Mar 31, 2022
cf4cb8e
Update playbook-ACTI_Block_Indicators_from_Incident_README.md
satyakidroid Mar 31, 2022
25f84ce
Update playbook-ACTI_Block_Indicators_severity_greater_or_equal_5_REA…
satyakidroid Mar 31, 2022
731c31b
Update playbook-ACTI_Create_Relationships_README.md
satyakidroid Mar 31, 2022
e754fc9
Update playbook-ACTI_IA_IR_Enrichment_README.md
satyakidroid Mar 31, 2022
85725f5
Update playbook-ACTI_Incident_Enrichment_README.md
satyakidroid Mar 31, 2022
b2dd783
Update playbook-ACTI_Indicator_Enrichment_README.md
satyakidroid Mar 31, 2022
2da74ad
Update playbook-ACTI_Vulnerability_Enrichment_README.md
satyakidroid Mar 31, 2022
85a2826
Update Playbook Name
nirmalneupane Mar 31, 2022
2cdd3b0
Update Playbook title
nirmalneupane Mar 31, 2022
1d5804d
Update playbook-ACTI_Create_Relationships.yml
nirmalneupane Mar 31, 2022
894d270
Update Playbook Name
nirmalneupane Mar 31, 2022
1e1dfb6
Description Updated
nirmalneupane Mar 31, 2022
1f17d93
Updating Playbook Description
nirmalneupane Mar 31, 2022
25ff47e
Update Playbook Description
nirmalneupane Mar 31, 2022
482101f
Updated Playbook Description.
nirmalneupane Mar 31, 2022
4fd34d1
Updated playbook config files
satyakidroid Apr 1, 2022
f4e8dfe
Minor Changes
satyakidroid Apr 1, 2022
ffa262d
Merge branch 'demisto:master' into ACTIIndicatorQuery_playbook
satyakidroid Apr 1, 2022
68e661c
Adding markdown postprocessing logic to download inline image links i…
nirmalneupane Apr 1, 2022
6cc79c0
Fixing errors
satyakidroid Apr 4, 2022
4310b7b
Errors Fixed : Try 2
satyakidroid Apr 4, 2022
248b894
Fixing ACTI Feed coverage issue
satyakidroid Apr 4, 2022
ed665ed
Fixing UT coverage for ACTIIndicatorQuery
satyakidroid Apr 4, 2022
f116adb
Fixing UTs and flake8 errors
satyakidroid Apr 5, 2022
2d4e4ba
Added UT for addBaseUrlToPartialPaths func
satyakidroid Apr 5, 2022
8f2abe1
UT added for domain and IP not found
ankitmordhwaj Apr 5, 2022
87709c4
UT for markdown postprocessing
ankitmordhwaj Apr 5, 2022
8dac89e
minor chnges
ankitmordhwaj Apr 5, 2022
42902d4
resolving flake8 errors
ankitmordhwaj Apr 5, 2022
1aad390
threatintelreport command & some minor bug fixed
satyakidroid Apr 7, 2022
a91aad5
Error fix : Try 1
satyakidroid Apr 7, 2022
8e23fd7
Minor changes
satyakidroid Apr 7, 2022
f8fef60
Fixing Nonetype object issue
satyakidroid Apr 7, 2022
7745a41
Fixed bug which was not creating relationship
satyakidroid Apr 12, 2022
f86a7e2
Removed Report Enrichment step in Incident Enrichment playbook
satyakidroid Apr 12, 2022
e4ecee5
Added user choice to block or not block indicators
satyakidroid Apr 13, 2022
146b3d8
Have added DeleteContext for user choice key and added ACTI Indicator…
satyakidroid Apr 13, 2022
58c1f1a
Added case when IA IR is not present, also have tweaked the playbook …
satyakidroid Apr 13, 2022
e16b5c1
Fixing Errors
satyakidroid Apr 13, 2022
145db5f
Update playbook-ACTI_Block_High_Severity_Indicators_README.md
satyakidroid Apr 13, 2022
ed496f8
Update playbook-ACTI_Block_Indicators_from_an_Incident_README.md
satyakidroid Apr 13, 2022
ad61926
Update playbook-ACTI_Create_Report-Indicator_Associations_README.md
satyakidroid Apr 13, 2022
e4c7813
Update playbook-ACTI_Incident_Enrichment_README.md
satyakidroid Apr 13, 2022
e6db9f4
Merge branch 'demisto:master' into ACTIIndicatorQuery_playbook
satyakidroid Apr 14, 2022
00133ba
Fixed bug which wasn't able to create relationship when there is 1 in…
satyakidroid Apr 14, 2022
f9b0295
Update playbook-ACTI_Incident_Enrichment_README.md
satyakidroid Apr 14, 2022
6ccae3e
Update playbook-ACTI_Block_Indicators_from_an_Incident.yml
satyakidroid Apr 19, 2022
7eeaa3b
Update playbook-ACTI_Block_High_Severity_Indicators.yml
satyakidroid Apr 19, 2022
3681152
Update playbook-ACTI_Vulnerability_Enrichment.yml
satyakidroid Apr 19, 2022
4f1caa8
Update playbook-ACTI_Vulnerability_Enrichment_README.md
satyakidroid Apr 19, 2022
95d291f
Update playbook-ACTI_Block_Indicators_from_an_Incident_README.md
satyakidroid Apr 19, 2022
7ced018
Update playbook-ACTI_Block_High_Severity_Indicators_README.md
satyakidroid Apr 19, 2022
2ea2887
Update playbook-ACTI_Create_Report-Indicator_Associations.yml
satyakidroid Apr 19, 2022
40fac03
Update playbook-ACTI_Create_Report-Indicator_Associations_README.md
satyakidroid Apr 19, 2022
212938a
Update playbook-ACTI_Create_Report-Indicator_Associations_README.md
satyakidroid Apr 19, 2022
3274e21
Update playbook-ACTI_Report_Enrichment.yml
satyakidroid Apr 19, 2022
f672a33
Update playbook-ACTI_Report_Enrichment_README.md
satyakidroid Apr 19, 2022
67b252a
Update playbook-ACTI_Indicator_Enrichment.yml
satyakidroid Apr 19, 2022
b24e805
Update playbook-ACTI_Indicator_Enrichment_README.md
satyakidroid Apr 19, 2022
d6858e7
Minor changes
satyakidroid Apr 20, 2022
f0edd0a
Merge branch 'contrib/iDefense_ACTIIndicatorQuery_playbook' into ACTI…
satyakidroid Apr 20, 2022
189e1f5
Merge branch 'demisto:master' into ACTIIndicatorQuery_playbook
satyakidroid Apr 20, 2022
3ec02e8
Update README.md
satyakidroid Apr 20, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Abstract of the long form report or blog body.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Analysis of the incident.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Conclusion of a intelligence report.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Indexing timestamp of report.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Last modified timestamp of report.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Last published timestamp of report.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description":"Recommendations based on incident summary and analysis.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Title of the report.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Numerical representation of severity from 1 to 5 with 1 being the least severe and 5 the most severe.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "List of associated critical intelligence requirement (CIR) types.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"group": 2,
"hidden": false,
"openEnded": false,
"description": "Unique ID of report.",
"associatedToAll": false,
"associatedTypes": [
"ACTI Intelligence Alert",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,5 +93,5 @@
},
"fileHashesPriority": null,
"expiration": 0,
"layout": "11efd063-3e83-4376-8572-d74a0846ab02"
}
"layout": "ACTI Intelligence Alert"
}
Original file line number Diff line number Diff line change
Expand Up @@ -81,5 +81,5 @@
},
"fileHashesPriority": null,
"expiration": 0,
"layout": "24adac2b-4fba-4760-89cd-adb8d3321aa3"
}
"layout": "ACTI Intelligence Report"
}
160 changes: 109 additions & 51 deletions Packs/AccentureCTI/Integrations/ACTIIndicatorQuery/ACTIIndicatorQuery.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,13 @@
class Client(BaseClient):
def __init__(self, input_url: str, api_key: str, verify_certificate: bool, proxy: bool, endpoint="/rest/threatindicator/v0"):
base_url = urljoin(input_url, endpoint)
PACK_VERSION = get_pack_version()
DEMISTO_VERSION = demisto.demistoVersion()
DEMISTO_VERSION = f'{DEMISTO_VERSION["version"]}.{DEMISTO_VERSION["buildNumber"]}'
headers = {
"Content-Type": "application/json",
'auth-token': api_key
'auth-token': api_key,
"User-Agent": f"AccentureCTI Pack/{PACK_VERSION} Palo Alto XSOAR/{DEMISTO_VERSION}"
}
super(Client, self).__init__(base_url=base_url,
verify=verify_certificate,
Expand Down Expand Up @@ -364,12 +368,10 @@ def _enrich_analysis_result_with_intelligence(analysis_info, doc_search_client,

alerts, reports = _get_ia_for_indicator(indicator, doc_search_client)

if alerts is not None:
analysis_info['Intelligence Alerts'] = alerts if len(
alerts) > 0 else 'No Intelligence Alert has been linked to this indicator'
if reports is not None:
analysis_info['Intelligence Reports'] = reports if len(
reports) > 0 else 'No Intelligence Report has been linked to this indicator'
if alerts:
analysis_info['Intelligence Alerts'] = alerts
if reports:
analysis_info['Intelligence Reports'] = reports

return analysis_info

Expand Down Expand Up @@ -407,15 +409,73 @@ def _get_ia_for_indicator(indicator: str, doc_search_client: Client):
return intelligence_alerts, intelligence_reports


def fix_markdown(text):
def markdown_postprocessing(md_text: str) -> str:
''' Applies post processing steps to fix markdown content for XSOAR viewing
Arg: md_text, markdown text to work on
Returns: output with processed markdown'''

result = fix_markdown(md_text)
result = addBaseUrlToPartialPaths(result)
result = convert_inline_image_to_encoded(result)
return result


def fix_markdown(text: str) -> str:
'''Fix markdown formatting issues
Arg: Text - Markdown text to be fixed'
Returns: output - Markdown with fixed formatting'''

regex_header = r"([#]+)([^\/|\s]\w)"
subst_header = "\\1 \\2"
result = re.sub(regex_header, subst_header, text, 0)
return result


def addBaseUrlToPartialPaths(content: str) -> str:
'''
append intelgraph's base URL to partial markdown links
e.g. '/rest/files/download/...' => 'https://intelgraph.idefense.com/rest/files/download/...'
e.g. '/#/node/region/view/...' => 'https://intelgraph.idefense.com/#/node/region/view/...
'''

files = r"\(\s?(\/rest\/.*?)\)"
relative_links = r"\((\s?(/#.*?|#.*?))\)"

def add_ig(match):
match = match.group(1)
if match[0] == " ":
match = match[1:]
if match[0] == '/':
match = match[1:]
return f'(https://intelgraph.idefense.com/{match})'

content = re.sub(relative_links, add_ig, content)
content = re.sub(files, add_ig, content)
return content


def convert_inline_image_to_encoded(md_text: str) -> str:
''' Converts inline images in markdown to base64 encoded images
arg: md_text, markdown text
return: result updated markdown text'''
regex = r'(!\[[^\]]+\])\((https?://[^\)]+)\)'
matches = re.findall(regex, md_text)
encoded_images = []
params = demisto.params()
api_key = params.get('api_token')
if isinstance(api_key, dict):
api_key = api_key.get('password')

regex_url = r"\/?#\/"
subst_url = "https://intelgraph.idefense.com/#/"
output = re.sub(regex_url, subst_url, result, 0)
return output
for single_match in matches:
single_image_link = single_match[1]
single_image_name = single_match[0]
response = requests.get(single_image_link,
headers={"auth-token": api_key}).content
data = base64.b64encode(response).decode('ascii')
image_type = single_image_link.split(".")[-1]
encoded_images.append(f'{single_image_name}(data:image/{image_type};base64,{data})')
result = re.sub(regex, lambda match: encoded_images.pop(0), md_text, 0, re.MULTILINE)
return result


def getThreatReport_command(doc_search_client: Client, args: dict, reliability: DBotScoreReliability):
Expand All @@ -441,11 +501,8 @@ def _ia_ir_extract(Res: dict, reliability: DBotScoreReliability):
"""
"""
threat_types = Res.get('threat_types', '')
threattypes = ''
uuid = Res.get('uuid', '')
if threat_types:
for threat_type in threat_types:
threattypes = threattypes + '\n- ' + threat_type

context = {
'created_on': Res.get('created_on', 'NA'),
'display_text': Res.get('display_text', 'NA'),
Expand All @@ -454,28 +511,28 @@ def _ia_ir_extract(Res: dict, reliability: DBotScoreReliability):
'last_modified': Res.get('last_modified', 'NA'),
'last_published': Res.get('last_published', 'NA'),
'links': Res.get('links', 'NA'),
'threat_types': threattypes,
'threat_types': threat_types,
'title': Res.get('title', 'NA'),
'type': Res.get('type', 'NA'),
'uuid': uuid,
'analysis': fix_markdown(Res.get('analysis', 'NA')),
'analysis': markdown_postprocessing(Res.get('analysis', 'NA')),
'sources_external': Res.get('sources_external', 'NA')
}

type_of_report = Res.get('type', 'NA')
if 'intelligence_report' in type_of_report:
context['conclusion'] = fix_markdown(Res.get('conclusion', 'NA'))
context['summary'] = fix_markdown(Res.get('summary', 'NA'))
context['conclusion'] = markdown_postprocessing(Res.get('conclusion', 'NA'))
context['summary'] = markdown_postprocessing(Res.get('summary', 'NA'))
severity_dbot_score = Common.DBotScore.NONE
indicatortype = 'ACTI Intelligence Report'
iair_link: str = IR_URL + uuid
else:
severity_dbot_score = Res.get('severity', 'NA')
if severity_dbot_score != 'NA':
severity_dbot_score = _calculate_dbot_score(severity_dbot_score)
context['mitigation'] = fix_markdown(Res.get('mitigation', 'NA'))
context['mitigation'] = markdown_postprocessing(Res.get('mitigation', 'NA'))
context['severity'] = Res.get('severity', 'NA')
context['abstract'] = fix_markdown(Res.get('abstract', 'NA'))
context['abstract'] = markdown_postprocessing(Res.get('abstract', 'NA'))
attachment_links = Res.get('attachment_links', '')
fqlink: str = ''
if attachment_links:
Expand All @@ -487,49 +544,50 @@ def _ia_ir_extract(Res: dict, reliability: DBotScoreReliability):
indicatortype = 'ACTI Intelligence Alert'
iair_link = IA_URL + uuid
dbot_score = Common.DBotScore(indicator=uuid, indicator_type=DBotScoreType.CUSTOM,
integration_name='ACTI Threat Intelligence Report',
integration_name='ACTI Indicator Query',
score=severity_dbot_score, reliability=reliability)
custom_indicator = Common.CustomIndicator(indicator_type=indicatortype, dbot_score=dbot_score,
value=uuid, data=context, context_prefix='IAIR')
return custom_indicator, iair_link


def main():
params = demisto.params()
api_key = params.get('api_token')
if isinstance(api_key, dict): # integration version >=3.2.0
api_key = api_key.get('password')
base_url = urljoin(params.get('url', ''))
reliability = params.get('integrationReliability', 'B - Usually reliable')

if DBotScoreReliability.is_valid_type(reliability):
reliability = DBotScoreReliability.get_dbot_score_reliability_from_str(reliability)
params = demisto.params() # pragma: no cover
api_key = params.get('api_token') # pragma: no cover
if isinstance(api_key, dict): # pragma: no cover # integration version >=3.2.0
api_key = api_key.get('password') # pragma: no cover
base_url = urljoin(params.get('url', '')) # pragma: no cover
reliability = params.get('integrationReliability', 'B - Usually reliable') # pragma: no cover

if DBotScoreReliability.is_valid_type(reliability): # pragma: no cover
reliability = DBotScoreReliability.get_dbot_score_reliability_from_str(reliability) # pragma: no cover
else:
Exception("ACTI error: Please provide a valid value for the Source Reliability parameter")
Exception("ACTI error: Please provide a valid value for the Source Reliability parameter") # pragma: no cover

commands = {
commands = { # pragma: no cover
'url': url_command,
'ip': ip_command,
'domain': domain_command,
'acti-get-ioc-by-uuid': uuid_command
}
verify_certificate = not params.get('insecure', False)
proxy = params.get('use_proxy', False)

try:
command = demisto.command()
client = Client(base_url, api_key, verify_certificate, proxy, endpoint=ENDPOINTS['threatindicator'])
document_search_client = Client(base_url, api_key, verify_certificate, proxy, endpoint=ENDPOINTS['document'])
demisto.debug(f'Command being called is {command}')
if command == 'test-module':
return_results(test_module(client))
elif command == 'acti-getThreatIntelReport':
return_results(getThreatReport_command(document_search_client, demisto.args(), reliability))
elif command in commands:
return_results(commands[command](client, demisto.args(), reliability, document_search_client))

except Exception as e:
return_error(f'Failed to execute {demisto.command()} command.\nError:\n{str(e)}')
verify_certificate = not params.get('insecure', False) # pragma: no cover
proxy = params.get('use_proxy', False) # pragma: no cover

try: # pragma: no cover
command = demisto.command() # pragma: no cover
client = Client(base_url, api_key, verify_certificate, proxy, endpoint=ENDPOINTS['threatindicator']) # pragma: no cover
document_search_client = Client(base_url, api_key, verify_certificate, # pragma: no cover
proxy, endpoint=ENDPOINTS['document']) # pragma: no cover
demisto.debug(f'Command being called is {command}') # pragma: no cover
if command == 'test-module': # pragma: no cover
return_results(test_module(client)) # pragma: no cover
elif command == 'acti-getThreatIntelReport': # pragma: no cover
return_results(getThreatReport_command(document_search_client, demisto.args(), reliability)) # pragma: no cover
elif command in commands: # pragma: no cover
return_results(commands[command](client, demisto.args(), reliability, document_search_client)) # pragma: no cover

except Exception as e: # pragma: no cover
return_error(f'Failed to execute {demisto.command()} command.\nError:\n{str(e)}') # pragma: no cover


if __name__ in ('__main__', '__builtin__', 'builtins'):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ script:
description: The actual score.
type: String

dockerimage: demisto/python3:3.10.1.27636
dockerimage: demisto/python3:3.10.4.28442
feed: false
isfetch: false
longRunning: false
Expand Down
Loading