Skip to content

Commit

Permalink
Remove hardcoded ECS version
Browse files Browse the repository at this point in the history
Update GHA workflow to only work on official wazuh-indexer repo

Add PR generation tool description at main ECS README
  • Loading branch information
QU3B1M committed Jan 7, 2025
1 parent 14d2184 commit 67a71bd
Show file tree
Hide file tree
Showing 5 changed files with 83 additions and 20 deletions.
4 changes: 3 additions & 1 deletion .github/workflows/generate-ecs-mappings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,16 @@ jobs:
- name: Set up Docker Compose
run: sudo apt-get install docker-compose

- name: Generate PR to wazuh-indxer-plugins
- name: Generate PR to wazuh-indexer-plugins
if: github.repository == 'wazuh/wazuh-indexer'
env:
GITHUB_TOKEN: ${{ secrets.ACTION_TOKEN }}
run: |
bash ecs/scripts/generate-pr-to-plugins.sh \
-b ${{ steps.branch-name.outputs.branch }} \
-o ../ecs-templates
- name: Upload artifact
if: github.repository == 'wazuh/wazuh-indexer'
uses: actions/upload-artifact@v4
with:
name: ecs-templates
Expand Down
52 changes: 49 additions & 3 deletions ecs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ This script generates the ECS mappings for the Wazuh indices.

### Requirements

- [Docker Desktop](https://docs.docker.com/desktop/setup/install/linux/)
> Other option is to install the [docker-compose plugin](https://docs.docker.com/compose/install/#scenario-two-install-the-docker-compose-plugin).
- [Docker Compose](https://docs.docker.com/compose/install/)

### Folder structure

Expand All @@ -27,7 +26,7 @@ There is a folder for each module. Inside each folder, there is a `fields` folde
A new `mappings` folder will be created inside the module folder, containing all the generated files.
The files are versioned using the ECS version, so different versions of the same module can be generated.
For our use case, the most important files are under `mappings/v8.11.0/generated/elasticsearch/legacy/`:
For our use case, the most important files are under `mappings/<ECS_VERSION>/generated/elasticsearch/legacy/`:
- `template.json`: Elasticsearch compatible index template for the module
- `opensearch-template.json`: OpenSearch compatible index template for the module
Expand Down Expand Up @@ -75,6 +74,53 @@ Each module contains a Python script to generate events for its module. The scri
The script will generate a JSON file with the events, and will also ask whether to upload them to the indexer. If the upload option is selected, the script will ask for the indexer URL and port, credentials, and index name.
The script uses log file. Check it out for debugging or additional information.
---
### Automatic PR creation tool
The `generate-pr-to-plugins.sh` script found in the `ecs/scripts` folder is a tool that detects modified ECS modules, generates new templates, commits the changes to a target repository, and creates or updates a pull request.
#### Requirements
- Docker Compose
- GitHub CLI (`gh`)
#### Usage
To use the script, run the following command:
```sh
./update-ecs-templates.sh -t <GITHUB_TOKEN>
```
**Options**
- `-b <BRANCH_NAME>`: (Optional) Branch name to create or update the pull request. Default is current branch.
- `-t <GITHUB_TOKEN>`: (Optional) GitHub token to authenticate with the GitHub API. If not provided, the script will use the `GITHUB_TOKEN` environment variable.
#### Script Workflow
1. **Validate Dependencies**
- Checks if the required commands (`docker`, `docker-compose`, and `gh`) are installed.
2. **Detect Modified Modules**
- Fetches and extracts modified ECS modules by comparing the current branch with the base branch.
- Identifies relevant ECS modules that have been modified.
3. **Run ECS Generator**
- Runs the ECS generator script for each relevant module to generate new ECS templates.
4. **Clone Target Repository**
- Clones the target repository (`wazuh/wazuh-indexer-plugins`) if it does not already exist.
- Configures Git and GitHub CLI with the provided GitHub token.
5. **Commit and Push Changes**
- Copies the generated ECS templates to the appropriate directory in the target repository.
- Commits and pushes the changes to the specified branch.
6. **Create or Update Pull Request**
- Creates a new pull request or updates an existing pull request with the modified ECS templates.
#### References
- [ECS repository](https://github.com/elastic/ecs)
Expand Down
5 changes: 4 additions & 1 deletion ecs/generator/images/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
FROM python:3.10

# Define the version as a build argument
ARG ECS_VERSION=v8.11.0

# Update the package list and upgrade all packages
RUN apt-get update && \
apt-get upgrade -y && \
Expand All @@ -9,7 +12,7 @@ RUN apt-get update && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
# Clone elastic ECS repository and install required Python libraries
git clone https://github.com/elastic/ecs.git -b v8.11.0 --depth 1 && \
git clone https://github.com/elastic/ecs.git -b ${ECS_VERSION} --depth 1 && \
pip install -r ecs/scripts/requirements.txt && \
# Create the directory for the ecs definitions (this will be used as a volume)
mkdir -p /source/ecs
Expand Down
2 changes: 1 addition & 1 deletion ecs/generator/mapping-generator.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ services:
volumes:
- ${REPO_PATH:-.}/ecs:/source/ecs
environment:
- ECS_MODULE=${ECS_MODULE:-default_module}
- ECS_MODULE=${ECS_MODULE}
40 changes: 26 additions & 14 deletions ecs/scripts/generate-pr-to-plugins.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
#!/usr/bin/env bash

# Constants
MAPPINGS_SUBPATH="mappings/v8.11.0/generated/elasticsearch/legacy/template.json"
ECS_VERSION=${ECS_VERSION:-v8.11.0}
MAPPINGS_SUBPATH="mappings/${ECS_VERSION}/generated/elasticsearch/legacy/template.json"
TEMPLATES_PATH="plugins/setup/src/main/resources/"
PLUGINS_REPO="wazuh/wazuh-indexer-plugins"
CURRENT_PATH=$(pwd)
Expand All @@ -26,7 +27,7 @@ validate_dependencies() {
done
}

fetch_and_extract_modules() {
detect_modified_modules() {
echo
echo "---> Fetching and extracting modified ECS modules..."
git fetch origin +refs/heads/master:refs/remotes/origin/master
Expand Down Expand Up @@ -105,15 +106,15 @@ clone_target_repo() {
commit_and_push_changes() {
echo
echo "---> Committing and pushing changes to ${PLUGINS_REPO} repository..."
git ls-remote --exit-code --heads origin "$branch_name" >/dev/null 2>&1
git ls-remote --exit-code --heads origin "$BRANCH_NAME" >/dev/null 2>&1
EXIT_CODE=$?

if [[ $EXIT_CODE == '0' ]]; then
git checkout "$branch_name"
git pull origin "$branch_name"
git checkout "$BRANCH_NAME"
git pull origin "$BRANCH_NAME"
else
git checkout -b "$branch_name"
git push --set-upstream origin "$branch_name"
git checkout -b "$BRANCH_NAME"
git push --set-upstream origin "$BRANCH_NAME"
fi

echo "Copying ECS templates to the plugins repository..."
Expand Down Expand Up @@ -154,7 +155,7 @@ create_or_update_pr() {
local title
local body

existing_pr=$(gh pr list --head "$branch_name" --json number --jq '.[].number')
existing_pr=$(gh pr list --head "$BRANCH_NAME" --json number --jq '.[].number')
# Format modules
modules_title=$(IFS=", "; echo "${relevant_modules[*]}")
modules_body=$(printf -- '- %s\n' "${relevant_modules[@]}")
Expand All @@ -168,7 +169,7 @@ create_or_update_pr() {
--title "$title" \
--body "$body" \
--base master \
--head "$branch_name"
--head "$BRANCH_NAME"
else
echo "PR already exists: $existing_pr. Updating the PR..."
gh pr edit "$existing_pr" \
Expand All @@ -178,8 +179,8 @@ create_or_update_pr() {
}

usage() {
echo "Usage: $0 -b <branch_name> -t <GITHUB_TOKEN>"
echo " -b <branch_name> Branch name to create or update the PR."
echo "Usage: $0 -b <BRANCH_NAME> -t <GITHUB_TOKEN>"
echo " -b [BRANCH_NAME] (Optional) Branch name to create or update the PR. Default: current branch."
echo " -t [GITHUB_TOKEN] (Optional) GitHub token to authenticate with GitHub API."
echo " If not provided, the script will use the GITHUB_TOKEN environment variable."
exit 1
Expand All @@ -189,7 +190,7 @@ main() {
while getopts ":b:t:o:" opt; do
case ${opt} in
b )
branch_name=$OPTARG
BRANCH_NAME=$OPTARG
;;
t )
GITHUB_TOKEN=$OPTARG
Expand All @@ -206,12 +207,23 @@ main() {
;;
esac
done
if [ -z "$branch_name" ] || [ -z "$GITHUB_TOKEN" ]; then

if [ -z "$BRANCH_NAME" ]; then
# Check if we are in a Git repository
if git rev-parse --is-inside-work-tree > /dev/null 2>&1; then
BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD)
else
echo "Error: You are not in a Git repository." >&2
exit 1
fi
fi

if [ -z "$BRANCH_NAME" ] || [ -z "$GITHUB_TOKEN" ]; then
usage
fi

validate_dependencies
fetch_and_extract_modules
detect_modified_modules
run_ecs_generator # Exit if no changes on relevant modules.
clone_target_repo
commit_and_push_changes # Exit if no changes detected.
Expand Down

0 comments on commit 67a71bd

Please sign in to comment.