Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wazuh packages redesign tier 2 #2904

Open
9 of 12 tasks
havidarou opened this issue Apr 8, 2024 · 1 comment
Open
9 of 12 tasks

Wazuh packages redesign tier 2 #2904

havidarou opened this issue Apr 8, 2024 · 1 comment
Assignees
@damarisg
Labels
level/objective type/enhancement Enhancement issue

Comments

@havidarou
Copy link
Member

havidarou commented Apr 8, 2024 •
edited by juliamagan
Loading

Description

Wazuh-packages (QA, Agent, CppServer)

  • Clean up everything
  • Add support to PPC packages (Agent)

Requirements and Restrictions

  • Wazuh packages redesign tier 1 requirements should be taken into account
  • The new repository must include QA related code and tools
    • New Jenkins should be used for everything QA related after 5.0
    • Current wazuh-packages repository is deprecated after 4.10
    • Current wazuh-jenkins repository is deprecated after 4.10
    • Current wazuh-tools repository is deprecated after 4.10: https://github.com/wazuh/wazuh-qa-automation/issues/417
    • Migrate scripts and pipelines that sync GitHub projects with GitHub.(QA)
    • Identify existing pipelines and migrate those that continue to provide coverage to Wazuh 4.x. (QA)
  • The release procedure should be improved by:
    - Improving repository metadata generation (avoid full metadata sync)
    - Reducing execution time
    - Incorporating backups and storage
    - Mirroring production bucket (release to the mirror, sync main with the mirror, restore main with the mirror)
  • The following will be migrated to the new DevOps repositories (both packages and automation):
    - OVA
    - AMI
    - Wazuh unattended scripts
    - Wazuh Puppet
    - Wazuh Docker
    - Wazuh Kubernetes
    - Wazuh Ansible
    - Training environment
    - Demo environment
  • Add support to PPC packages (Agent)
  • The following installers will be migrated from wazuh-packages repository:
    Agent installers:
    - HP-UX
    - Solaris Intel/SPARC
    - AIX

Auditor

  • Involved: DevOps, QA, Agent, CppServer
  • esponsible: @wazuh/devel-qa-div3
  • DRI name: @davidjiglesias
  • Objective: Wazuh packages redesign tier 2

Planning

Precondition

Previous tier

Dependencies

Spike
Tasks

QA

Further improvements are required

Move to 4.10.1

DevOps

Agent

Conclusion (WIP)

QA

Release procedure
  • S3 sign check: S3 sign checking now works in multi-threaded mode, where each thread runs in a separate VM to handle its errors (destroying the VM and leaving a log of the error). It helps achieve asynchronous independence from the executions in S3_sign_check.
  • Reducing execution time: There's no way to speed up DEB metadata generation without creating bigger problems that could compromise the process.
  • Allow release-tool.py execution on AWS: It allows the launch tool to run on AWS infrastructure (4vcpu and 8GB).
    In addition, it was adapted to local execution and in local containers, providing a single entry point for the tool. It balanced the increase in time in metadata generation.
    Stage v4.9.0-beta2 Current implementation on AWS
    RPM - Download 52m 7m
    RPM - Metadata and upload - 4m20s
    RPM - Metadata 20s -
    RPM - Upload 40s -
    DEB - Download 49m 5m
    DEB - Metadata and upload - 46m
    DEB - Metadata 22m -
    DEB - Upload 40s -
  • Improving repository metadata generation: Added new documentation on how to modify recipe files and regenerate metadata without adding files to the repository, added new use cases such as Stage support, release protocol, and metadata regeneration, improved requirements, and removed redundant information.
  • Incorporating backups and storage: It provides an optional backup mechanism and a precondition feature that allows the execution of ordered tasks and allows some operations before starting with the core release procedure.
  • Mirroring production bucket (release to the mirror, sync main with the mirror, restore main with the mirror): Currently, we sync prod into prod-mirror to guarantee that prod-mirror is a mirror and deploy packages into prod-mirror. After that, the backup prod by into the directory (s3://packages.wazuh.com/Backups/Backup_{timestamp}/{major}.x) and we sync prod-mirror into prod. Finally, we AWS S3-to-S3 copy, without localhost intervention, only deltas, and then we invalidate the cache of the prod.
Restore support to PPC packages (Agent - deb ppc64el, rpm ppc64le)

The changes consisted of:

  • Update Wazuh documentation.
  • Adapt release procedures to support PPC packages (package generation, package signing and verification, and release tool deployment).
  • Update release issues (tests).
The new repository must include QA related code and tools
  • Defines a private repository: “wazuh-qa-automation”.
  • Currently, we have two branches: 4.10.0 and the main.
  • The new repository includes:
    • New templates for reporting bugs, requesting new support, and creating PRs.
    • New README with a use guide for the tools migrated.
    • Apply new structure and good practices.
    • The 4.10.0 and 5.0.0 branches have the code migrated.
    • For 4.10.0
      • Workflow
        • s3_sign_check: Adds the requirement for a GitHub PAT that must be set using the GITHUB_TOKEN environment variable.
        • dtt1: Adds two steps to workflow files. Now, the token must be manually filled in before execution.
      • Adapted workflows that use the Allocator module that are on wazuh-automation.
      • Identified existing pipelines to migrate.
      • Migrate JobFlow tools
      • Migrated Provision module
      • Migrated release procedures
      • Migrated new Jenkins's code
      • Migrated Tests module and DTT workflows
      • Migrated the environments provisioning module
      • Migrated the System tests - All
      • Migrated the End to End tests - Only VD
      • Migrated scripts and pipelines that sync GitHub projects with GitHub.
      • Migrated pipelines of:
        • Test_registration
        • Test Register Tier
        • Test Service
        • Test_upgrade pipeline to the new Jenkins
        • Test_upgrade_tier pipeline to the new Jenkins
        • Test_stress pipeline to the new Jenkins.
        • Test_stress_gh pipeline to the new Jenkins.
        • Test_stress_Vagrant pipeline to the new Jenkins.
        • Test_stress_tier pipeline to the new Jenkins.
        • Procedure_ecs_task pipeline to the new Jenkins.
        • Procedure_ecs_task_tier pipeline to the new Jenkins.
        • procedure_cicd_release_issues pipeline to the new Jenkins.
        • Procedure_gh_project_syncup pipeline to the new Jenkins.
      • Delete any dependency of wazuh-packages repository .
      • Delete any dependency of wazuh-tools repository .
      • Delete any dependency of wazuh-packages repository .
      • Delete any dependency of qa-integration-framework repository .
      • Delete any dependency of qa-system-framework repository .
      • Delete any dependency of wazuh-packages repository .
The installers migrated from wazuh-packages repository
  • DevOps GHA was adapted with a best-effort approach but a unique interface should be defined and populated among all subsystems to be releasable QA release procedures were too related to Wazuh Packages Redesign Tier 1, difficult to modularize and extend. Following aspects are some of them:
    • Generate packages only consider deliverable as files. This is not true for i.e AMI and Docker
    • Generate packages only consider one file per target. This is not true for installation-assistant
    • Generate packages set several constraints on GHA that are not documented: id, id as part of the name, checksum
    • Sign check only allows tagging files that imply an upload procedure. This is not necessary when the file does not need a signature
OUT OF SCOPE: "Packages Redesign Tier 2"

  • We are working on https://github.com/wazuh/wazuh-qa-automation/issues/42 which proposes new processes and practices to work.

  • The 5.0.0 branch should remove everything that Migrate has deprecated.

  • Once the target tasks are complete, we will work on implementing some improvements, which are:

    Release Procedures

    Medium priority improvements:

    • tools/release/src/**/docker/Dockerfile_tests: Each module's docker test image.
      We should re-evaluate if it's necessary to run each module's unit tests on a docker image, noting that it consumes time and may not add much value to the development itself.
    • .github/workflows: Main GHA workflows directory
      • Currently, all the Release Procedures modules' Unit Tests are being executed on GHA and running on Docker, which means that we build a Docker image and run it on each GHA, and it is resources-consuming.
      • All the GHA Workflows can be simplified in just one workflows that runs all the UTs (but first, it would be great if we get rid of the Docker build)
    Testing

    High-priority improvements:

    • tools/testing/src/testing/testing.py: Main testing module file
      • The module's parameters are strongly attached to DTT, it must be more generic to re-use it on different suites.
      • It expects the target and dependencies inventories to be on a specific path with a desired format, so it won't detect the target system correctly if the inventory is in a custom directory.
    • tools/testing/src/testing/playbooks/test.yml Test execution playbook
      The test path is generated "dynamically" but it depends completely on the DTT format
    • tests/test_functional/test_system/test_deployability/workflows: DTT workflows directory
      • The workflows are outdated, and its using the legacy allocator path, it must be updated to use them.
      • Wazuh version on the Workflows is hardcoded using an outdated value.
      • Add parametrization on the Workflows where possible.
    • tools/testing/tests: Testing tool Unit Tests directory
      There are no Unit Tests developed, we must create a set of tests to ensure its correct functionality
      Related issues:

    Medium priority improvements:

    • tests/test_functional/test_system/test_deployability/tests: Main DTT tests directory
      The test helpers have a lot of duplicated code, and there are several wrongly handled conditional flows. We could include unit tests for the helpers to validate its correct behavior
    JobFlow

    Medium priority improvements:

    • tools/jobflow/README.md: Main documentation file of the JobFlow tool
      Currently, the documentation is DTT1-specific, it does not help the user to understand what the tool does, and how it can be used for different cases.

    Low priority improvements:

    • tools/jobflow/tests: Unit Tests suite
      This suite uses Python's default test runner UnitTests utilities and pytest as the runner, it could be improved by only using pytest and its utilities, which is recommended.
      Related issues:
    • tools/jobflow/examples: Workflows examples directory
      We must add more Workflows as examples of different use cases.

    Other improvements could be found here: Jobflow engine - Findings and recommendations wazuh-qa#5044

    Provision

    Medium priority improvements:

    • tools/provision/README.md: Main documentation file of the Provision tool
      This README is strongly related to the JobFlow, it could be improved detailing mainly the standalone execution and having the JobFlow workflow as a secondary use.

    Low-priority improvements:

    • tools/provision/src/provision/playbooks/wazuh: Wazuh-related playbooks
      This directory could be re-structured, probably the non install/uninstall actions (register, services, set repo) could be separated on a different folder as these actions are not specifically related to an installation type.
    • tools/provision/tests: Unit Tests suite
      This suite uses Python's default test runner UnitTests utilities and pytest as the runner, it could be improved by only using pytest and its utilities, which is recommended.
      Related issues:
    Jenkins

    Low-priority improvements:

    • tools/jenkins/tests: tool's Unit Tests directory
      Must implement more Unit Tests for this tool
    • tools/jenkins/src/job-builder: Jenkins pipelines as a code
      Add a README.md detailing this folder, it could be implemented here or in the main Jenkins src directory (tools/jenkins/src), in the second case, it should also includes a summary about bootstrap.

Deliverables

4.10.0 Alpha 2

QA

DevOps

Agent

4.10.0 - Alpha 3

QA

Agent

4.10.1 - Alpha 1

QA
@rauldpm
Copy link
Member

rauldpm commented Apr 8, 2024

Issue content migrated from https://github.com/wazuh/internal-devel-requests/issues/993

@vikman90 vikman90 added this to Roadmap Apr 8, 2024
@vikman90 vikman90 moved this to Draft in Roadmap Apr 8, 2024
@rauldpm rauldpm mentioned this issue May 2, 2024
Closed
6 tasks
@vikman90 vikman90 mentioned this issue May 3, 2024
Closed
@davidjiglesias davidjiglesias closed this as not planned Won't fix, can't repro, duplicate, stale Jun 26, 2024
@github-project-automation github-project-automation bot moved this from Draft to Done in Roadmap Jun 26, 2024
@davidjiglesias davidjiglesias moved this from Done to Backlog in Roadmap Jun 26, 2024
@davidjiglesias davidjiglesias moved this from Backlog to Draft in Roadmap Jun 26, 2024
@wazuhci wazuhci moved this from Draft to Done in Roadmap Jun 26, 2024
@wazuhci wazuhci moved this from Done to Backlog in Roadmap Jun 27, 2024
@QU3B1M QU3B1M mentioned this issue Jun 28, 2024
Closed
8 tasks
@wazuhci wazuhci moved this to Backlog in Release 4.10.0 Jul 1, 2024
@damarisg damarisg self-assigned this Jul 3, 2024
@wazuhci wazuhci moved this from Backlog to In progress in Release 4.10.0 Jul 3, 2024
@vikman90 vikman90 mentioned this issue Jul 9, 2024
Closed
This was referenced Jul 10, 2024
Closed
Closed
Closed
@pcapellan pcapellan mentioned this issue Jul 15, 2024
Closed
2 tasks
@jnasselle jnasselle mentioned this issue Jul 16, 2024
Closed
@wazuhci wazuhci moved this from Backlog to In progress in Roadmap Aug 2, 2024
This was referenced Aug 6, 2024
Closed
Closed
Closed
Closed
This was referenced Aug 7, 2024
Closed
Closed
This was referenced Aug 8, 2024
Closed
Closed
This was referenced Aug 8, 2024
Closed
Closed
Closed
Closed
@damarisg damarisg mentioned this issue Sep 11, 2024
Closed
8 tasks
@c-bordon c-bordon mentioned this issue Sep 12, 2024
Closed
10 tasks
@rauldpm rauldpm mentioned this issue Oct 11, 2024
Closed
@teddytpc1 teddytpc1 mentioned this issue Oct 21, 2024
Closed
1 task
@wazuhci wazuhci removed this from Release 4.10.0 Nov 19, 2024
This was referenced Nov 26, 2024
Closed
Closed
@guidomodarelli guidomodarelli mentioned this issue Dec 9, 2024
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@damarisg damarisg

Labels
level/objective type/enhancement Enhancement issue
Projects
Roadmap
Status: In progress
Milestone
No milestone
Development

No branches or pull requests
5 participants
@damarisg @havidarou @rauldpm @davidjiglesias and others