Merge PR #4469 From @phantinuss - Add Release Packages

chore: add workflows, scripts and documentation for release packages

---------

Co-authored-by: Nasreddine Bencherchali <[email protected]>

.github/workflows/release.yml

@@ -0,0 +1,63 @@
+on:
+  push:
+    tags:
+      - 'r*'
+
+name: Create Release
+
+jobs:
+  build:
+    name: Create Release
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+      - name: Generate Changelog
+        run: |
+          prev_tag=$(git for-each-ref --sort=creatordate --format '%(refname:lstrip=2)' refs/tags | grep ^r | tail -2 | head -1)
+          curr_tag=$(git for-each-ref --sort=creatordate --format '%(refname:lstrip=2)' refs/tags | grep ^r | tail -1)
+          echo "Previous tag: ${prev_tag}"
+          echo "Current tag: ${curr_tag}"
+          if [[ $(git log --pretty=%B ${prev_tag}..${curr_tag} | grep -E '^\s*new: ' -c) -gt 0 ]]; then echo "### New Rules" > changes.txt; fi
+          git log --pretty=%B ${prev_tag}..${curr_tag} | grep -E '^\s*new: ' | sort | sed -e 's%^% - %' >> changes.txt
+          if [[ $(git log --pretty=%B ${prev_tag}..${curr_tag} | grep -E '^\s*update: ' -c) -gt 0 ]]; then echo "### Updated Rules" >> changes.txt; fi
+          git log --pretty=%B ${prev_tag}..${curr_tag} | grep -E '^\s*update: ' | sort | sed -e 's%^% - %' >> changes.txt
+          if [[ $(git log --pretty=%B ${prev_tag}..${curr_tag} | grep -E '^\s*fix: ' -c) -gt 0 ]]; then echo "### Fixed Rules" >> changes.txt; fi
+          git log --pretty=%B ${prev_tag}..${curr_tag} | grep -E '^\s*fix: ' | sort | sed -e 's%^% - %' >> changes.txt
+          git log --pretty=%B ${prev_tag}..${curr_tag} | grep -oP 'Merge PR #\d+ from \K(@\S+)' | sort -u > authors_raw.txt
+          git log --pretty=%B ${prev_tag}..${curr_tag} | grep -oP "Co-authored-by: \K.*(?= <)" | sort -u | sed -e 's%^%@%' >> authors_raw.txt
+          LC_ALL=en_US.UTF-8 sort -u authors_raw.txt | grep -v 'dependabot\[bot\]' > authors.txt
+          cat changes.txt >> changelog.txt
+          echo "" >> changelog.txt
+          echo "### Acknowledgement" >> changelog.txt
+          echo "Thanks to $(perl -pe 's%\n%, %' authors.txt | sed 's%, $%%') for their contribution to this release" >> changelog.txt
+          echo "" >> changelog.txt
+          echo "" >> changelog.txt
+          echo "### Which Sigma rule package should I use?" >> changelog.txt
+          echo "A detailed explanation can be found in the [Releases.md](Releases.md) file. If you are new to Sigma, we recommend starting with the \"Core\" ruleset." >> changelog.txt
+          cat changelog.txt
+      - name: Build all release packages
+        run: |
+          python3 tests/sigma-package-release.py --min-status test --min-level high --rule-types generic --outfile sigma_core_${{ github.ref_name }}.zip
+          python3 tests/sigma-package-release.py --min-status test --min-level medium --rule-types generic --outfile sigma_core+_${{ github.ref_name }}.zip
+          python3 tests/sigma-package-release.py --min-status experimental --min-level medium --rule-types generic --outfile sigma_core++_${{ github.ref_name }}.zip
+          python3 tests/sigma-package-release.py --min-status experimental --min-level medium --rule-types et --outfile sigma_emerging_threats_addon_${{ github.ref_name }}.zip
+          python3 tests/sigma-package-release.py --min-status experimental --min-level medium --rule-types generic et --outfile sigma_all_rules_${{ github.ref_name }}.zip
+      - name: Create Release with Assets
+        id: create_release
+        uses: softprops/action-gh-release@v1
+        with:
+          tag_name: ${{ github.ref }}
+          name: Release ${{ github.ref_name }}
+          body_path: changelog.txt
+          token: ${{ secrets.GITHUB_TOKEN }}
+          draft: false
+          prerelease: false
+          files: |
+            sigma_core_${{ github.ref_name }}.zip
+            sigma_core+_${{ github.ref_name }}.zip
+            sigma_core++_${{ github.ref_name }}.zip
+            sigma_emerging_threats_addon_${{ github.ref_name }}.zip
+            sigma_all_rules_${{ github.ref_name }}.zip

Releases.md

@@ -0,0 +1,75 @@
+This following document describes the different types of rule packages provided with every release.
+
+## Package Introduction
+
+The rule packages provided with every release are split based on the [status](https://github.com/SigmaHQ/sigma-specification/blob/main/Sigma_specification.md#status-optional), [level](https://github.com/SigmaHQ/sigma-specification/blob/main/Sigma_specification.md#level) and [type](https://medium.com/sigma-hq/sigma-rule-repository-enhancements-new-folder-structure-rule-types-30adb70f5e10) of a sigma rule.
+
+There are currently 3 main rule types provided in the sigma repository:
+
+- **core/generic**: Rules that match on attacker techniques. These rules are timeless and often match on new threats.
+- **emerging-threats/ET**: Rules that match on patterns of specific threat actors or exploits. High signal to noise ratio but will decrease in relevance over time.
+- **threat-hunting/TH**: Rules that should not be run for alerting but are interesting in giving detection ideas or hunt for suspicious activity inside an environment.
+
+### Package Overview
+
+name | status | level | type
+--- | --- | --- | ---
+[Core (Default)](#core-rules) | testing, stable | high, critical | core
+[Core+ (Rule Review needed)](#core-rules-1) | testing, stable | medium, high, critical | core
+[Core++ (Experimental)](#core-rules-2) | experimental, testing, stable | medium, high, critical | core
+[Emerging Threats AddOn Rules](#et-emerging-threats-addon-rules) | experimental, testing, stable | medium, high, critical | emerging threats
+[All rules](#all-rules) | experimental, testing, stable | medium, high, critical | core, emerging threats
+
+If you are new, best start with the `Core` Sigma package. It includes high quality rules of high confidence and relevance and should not produce many false positives.
+
+If your setup is working fine, you can add the `emerging threats` rules and start thinking about upgrading to `Core+` rules. If that is not enough and you like the pain, use the "all" rules package.
+
+### Defined Package
+
+#### Core Rules
+
+The `Core` Sigma package includes high quality rules of high confidence and relevance and should not produce many false positives.
+
+The selected rules are of level `high` or `critical`, which means matches are of high or critical importance. The rule status is `testing` or `stable`, which means the rule is at least of an age of half a year and no false positives were reported on it.
+
+The type is `core`, meaning the rules will match on attacker technique and generic suspicious or malicious behavior.
+
+#### Core+ Rules
+
+The plus in the `Core+` Sigma package stands for the addition of `medium` level rules. Those rules most often need additional tuning as certain applications, legitimate user behavior or scripts of an organization might be matched. Not every `medium` level rule is useful in every organization.
+
+#### Core++ Rules
+
+The `Core++` package additionally includes the rules of `experimental` status. These rules are bleeding edge. They are validated against the Goodlog tests available to the SigmaHQ project and reviewed by multiple detection engineers. Other than that they are pretty much untested at first. Use these if you want to be able to detect threats as early as possible at the cost of managing a higher threshold of false positives.
+
+Please report any false positives you find in the wild via our [github issue tracker](https://github.com/SigmaHQ/sigma/issues/new?assignees=&labels=False-Positive&projects=&template=false_positive_report.yml). After a grace period all `experimental` rules will eventually be promoted to status `test`.
+
+### Package AddOn's
+
+#### ET (Emerging Threats) AddOn Rules
+
+The `ET AddOn` Sigma package contains all of the `emerging threats` rules. These rules have a low false positive rate so that it already contains rules of status `experimental`. These rules target specific threats and are especially useful for current threats where maybe not much information is yet available. So we want to get them to you as fast as possible. The package is an `AddOn` so you can use it on top of whichever `Core` package is most useful to you.
+
+### All Rules
+
+> **Note**
+>
+> This package doesn't contain all rules
+
+This package includes all rules from level `medium` with a status of `experimental` and upwards including the `emerging threats` rules. Some heavy tuning is required when using this package.
+
+You'll notice that rules of level `low` and some other are omitted even from this the `All Rules` package. We do not recommend using any other types of rules to generate alerts except for those provided in these packages.
+
+### Create Your Own Custom Rule Package
+
+Releases are tagged using the format `r<ISO 8601 date>` (e.g. `r2023-12-24`).
+
+You can checkout any release version and create your own package using the [sigma-package-release](tests/sigma-package-release.py) script. Define the `status`, `level` and `type` of rules and the script generates a ZIP archive containing only those rules.
+
+e.g.
+
+```bash
+# python3 tests/sigma-package-release.py --min-status testing --levels high critical --types generic --outfile Sigma-custom.zip
+```
+
+You can either give `level` and `status` as a space separated list or using a minimum value. See `--help` for all options

tests/sigma-package-release.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Creates the Sigma release archive packages for different configurations
+
+EXAMPLE
+# python3 sigma-package-release.py --min-status testing --levels high critical --rule-types generic --outfile Sigma-standard.zip
+"""
+
+import os
+import sys
+import argparse
+import yaml
+import zipfile
+
+
+STATUS = ["experimental", "test", "stable"]
+LEVEL = ["informational", "low", "medium", "high", "critical"]
+RULES_DICT = {
+    "generic": "rules",
+    "rules": "rules",
+    "core": "rules",
+    "emerging-threats": "rules-emerging-threats",
+    "rules-emerging-threats": "rules-emerging-threats",
+    "et": "rules-emerging-threats",
+    "threat-hunting": "rules-threat-hunting",
+    "th": "rules-threat-hunting",
+    "rules-threat-hunting": "rules-threat-hunting"
+    }
+RULES = [x for x in RULES_DICT.keys()]
+
+def init_arguments(arguments: list) -> list:
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument('-o', '--outfile', help="Outputs the Sigma release package as ZIP archive", default="Sigma-standard.zip", required=True)
+    arg_status = parser.add_mutually_exclusive_group(required=True)
+    arg_status.add_argument('-s', '--statuses', nargs='*', choices=STATUS, help="Select status of rules")
+    arg_status.add_argument('-ms', '--min-status', nargs='?', choices=STATUS, help="Sets the minimum status of rules to select")
+    arg_level = parser.add_mutually_exclusive_group(required=True)
+    arg_level.add_argument('-l', '--levels', nargs='*', choices=LEVEL, help="Select level of rules")
+    arg_level.add_argument('-ml', '--min-level', nargs='?', choices=LEVEL, help="Sets the minimum level of rules to select")
+    parser.add_argument('-r', '--rule-types', choices=RULES, nargs='*', help="Select type of rules")
+    args = parser.parse_args(arguments)
+
+    if not args.outfile.endswith(".zip"):
+        args.outfile = args.outfile + ".zip"
+
+    if os.path.exists(args.outfile):
+        print("[E] '{}' already exists. Choose a different output file name.".format(args.outfile))
+        sys.exit(1)
+
+    if args.rule_types == None:
+        args.rule_types = ["generic"]
+        print("[I] -r/--rule-types not defined: Using \"generic\" by default")
+
+    if args.min_level != None:
+        i = LEVEL.index(args.min_level)
+        args.levels = LEVEL[i:]
+
+    if args.min_status != None:
+        i = STATUS.index(args.min_status)
+        args.statuses = STATUS[i:]
+
+    return args
+
+def select_rules(args: dict) -> list:
+    selected_rules = []
+
+    def yield_next_rule_file_path(rule_path: str) -> str:
+        for root, _, files in os.walk(rule_path):
+            for file in files:
+                if file.endswith('.yml'):
+                    yield os.path.join(root, file)
+
+    def get_rule_yaml(file_path: str) -> dict:
+        data = []
+
+        with open(file_path, encoding='utf-8') as f:
+            yaml_parts = yaml.safe_load_all(f)
+            for part in yaml_parts:
+                data.append(part)
+        return data
+
+    for rules_path_alias in args.rule_types:
+        rules_path = RULES_DICT[rules_path_alias]
+        for file in yield_next_rule_file_path(rule_path=rules_path):
+            rule_yaml = get_rule_yaml(file_path=file)
+            if len(rule_yaml) != 1:
+                print("[E] rule {} is a multi-document file and will be skipped".format(file))
+                continue
+
+            rule = rule_yaml[0]
+            if (rule["level"] in args.levels and
+                rule["status"] in args.statuses):
+                selected_rules.append(file)
+
+    return selected_rules
+
+def write_zip(outfile: str, selected_rules: list):
+    with zipfile.ZipFile(outfile, mode='a', compression=zipfile.ZIP_DEFLATED, compresslevel=9) as zip:
+        for rule_path in selected_rules:
+            zip.write(rule_path)
+    return
+
+def main(arguments: list) -> int:
+    args = init_arguments(arguments)
+
+    print("[I] Parsing and selecting rules, this will take some time...")
+    selected_rules = select_rules(args)
+    print("[I] Selected {} rules".format(len(selected_rules)))
+
+    write_zip(args.outfile, selected_rules)
+    print("[I] Written all rules to output ZIP file '{}'".format(args.outfile))
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:]))