[ACIX-453] Agent 6 release tasks support #30062

CelianR · 2024-10-11T15:27:10Z

What does this PR do?

Provides support for agent 6 tasks/release.py.
With worktrees, the agent_context utility can execute the task on a specific branch (usually the release_branch).

Note

Most of the changes are within tasks/release.py (you should hit Load Diff to see the changes).
Feel free to check out the documentation for worktrees.

Workflow

All the tooling is kept within the main branch of the agent. To run a release task, run inv release.<task> and specify the agent version to run it on a specific branch.

When executing such task, the task will check out the correct branch and will continue the execution given this context.

Note

Some tasks do not support multiple versions anymore.

Motivation

Describe how to test/QA your changes

Possible Drawbacks / Trade-offs

Additional Notes

agent-platform-auto-pr · 2024-11-27T12:58:38Z

Gitlab CI Configuration Changes

Modified Jobs

.check_pkg_size

  .check_pkg_size:
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    script:
    - ls -l $OMNIBUS_PACKAGE_DIR
    - if [[ "${ARCH}" == "amd64" ]]; then ls -l $OMNIBUS_PACKAGE_DIR_SUSE; fi
    - export failures=0
-   - export last_stable=$(inv release.get-release-json-value "last_stable::${MAJOR_VERSION}")
?                                                                                            -
+   - export last_stable=$(inv release.get-release-json-value "last_stable::${MAJOR_VERSION}"
+     --no-worktree)
    - "if [[ \"${ARCH}\" == \"amd64\" ]]; then ARCH_RPM_EXT=\"x86_64\"; else ARCH_RPM_EXT=\"\
      aarch64\"; fi\nfor flavor in ${FLAVORS}; do\n\n    if [[ \"${ARCH}\" == \"amd64\"\
      \ && \"$flavor\" != \"datadog-heroku-agent\" ]]; then\n      mkdir -p \"/tmp/stable/${flavor}/suse\"\
      \n      curl -sSL \"https://s3.amazonaws.com/yum.datadoghq.com/suse/stable/${MAJOR_VERSION}/${ARCH_RPM_EXT}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \ -o \"/tmp/stable/${flavor}/suse/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \n      set +e\n      inv package.compare-size --package-type \"${flavor} suse\
      \ rpm\" --last-stable \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\"\
      \ --new-package \"$OMNIBUS_PACKAGE_DIR_SUSE/${flavor}-${MAJOR_VERSION}.*.${ARCH_RPM_EXT}.rpm\"\
      \ --stable-package \"/tmp/stable/${flavor}/suse/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \n      failures=$((${failures}+$?))\n      set -e\n    fi\n\n    mkdir -p \"\
      /tmp/stable/${flavor}\"\n\n    curl -sSL \"https://s3.amazonaws.com/apt.datadoghq.com/pool/d/da/${flavor}_${last_stable}-1_${ARCH}.deb\"\
      \ -o \"/tmp/stable/${flavor}/${flavor}_${last_stable}-1_${ARCH}.deb\"\n\n    set\
      \ +e\n    inv package.compare-size --package-type \"${flavor} deb\" --last-stable\
      \ \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\" --new-package \"\
      $OMNIBUS_PACKAGE_DIR/${flavor}_${MAJOR_VERSION}*_${ARCH}.deb\" --stable-package\
      \ \"/tmp/stable/${flavor}/${flavor}_${last_stable}-1_${ARCH}.deb\"\n    failures=$((${failures}+$?))\n\
      \    set -e\n\n    if [[ \"$flavor\" != \"datadog-heroku-agent\" && ( \"${ARCH}\"\
      \ == \"amd64\" || \"$flavor\" != \"datadog-dogstatsd\") ]]; then\n      # We don't\
      \ build RPM packages for the heroku flavor\n      curl -sSL \"https://s3.amazonaws.com/yum.datadoghq.com/stable/${MAJOR_VERSION}/${ARCH_RPM_EXT}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \ -o \"/tmp/stable/${flavor}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\n\
      \      set +e\n      inv package.compare-size --package-type \"${flavor} rpm\"\
      \ --last-stable \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\" --new-package\
      \ \"$OMNIBUS_PACKAGE_DIR/${flavor}-${MAJOR_VERSION}.*.${ARCH_RPM_EXT}.rpm\" --stable-package\
      \ \"/tmp/stable/${flavor}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\n \
      \     failures=$((${failures}+$?))\n      set -e\n    fi\ndone\n"
    - if [ "${failures}" -ne "0" ]; then false; fi
    stage: pkg_metrics
    tags:
    - arch:amd64

.new-e2e_windows_a7_x86_64

  .new-e2e_windows_a7_x86_64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    artifacts:
      expire_in: 2 weeks
      paths:
      - $E2E_OUTPUT_DIR
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - WINDOWS_AGENT_VERSION=$(invoke agent.version) || exit $?; export WINDOWS_AGENT_VERSION
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_e2e.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_public_key_rsa > $E2E_PUBLIC_KEY_PATH
      || exit $?
    - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
      $AGENT_QA_E2E ssh_key_rsa > $E2E_PRIVATE_KEY_PATH || exit $?
    - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
    - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE client_id)
      || exit $?; export ARM_CLIENT_ID
    - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE token)
      || exit $?; export ARM_CLIENT_SECRET
    - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE tenant_id)
      || exit $?; export ARM_TENANT_ID
    - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE subscription_id)
      || exit $?; export ARM_SUBSCRIPTION_ID
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_GCP credentials_json > ~/gcp-credentials.json
      || exit $?
    - export GOOGLE_APPLICATION_CREDENTIALS=~/gcp-credentials.json
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_e2e_deps
    - deploy_windows_testing-a7
    rules:
    - if: ($CI_COMMIT_BRANCH == "main"  || $DEPLOY_AGENT == "true" || $RUN_E2E_TESTS
        == "on" || $DDR_WORKFLOW_ID != null) && $RUN_E2E_TESTS != "off"
    script:
-   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7")
?                                                                                        -
+   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7"
+     --no-worktree)
    - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
      -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
      --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
    stage: e2e
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
      E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
      E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
      E2E_PIPELINE_ID: $CI_PIPELINE_ID
      E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
      E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      EXTRA_PARAMS: --run "$E2E_MSI_TEST$"
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      SHOULD_RUN_IN_FLAKES_FINDER: 'true'
      TARGETS: ./tests/windows/install-test
      TEAM: windows-agent
      WINDOWS_AGENT_ARCH: x86_64

.new-e2e_windows_domain_test

  .new-e2e_windows_domain_test:
    before_script:
    - WINDOWS_AGENT_VERSION=$(invoke agent.version) || exit $?; export WINDOWS_AGENT_VERSION
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_e2e.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_public_key_rsa > $E2E_PUBLIC_KEY_PATH
      || exit $?
    - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
      $AGENT_QA_E2E ssh_key_rsa > $E2E_PRIVATE_KEY_PATH || exit $?
    - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
    - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE client_id)
      || exit $?; export ARM_CLIENT_ID
    - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE token)
      || exit $?; export ARM_CLIENT_SECRET
    - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE tenant_id)
      || exit $?; export ARM_TENANT_ID
    - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE subscription_id)
      || exit $?; export ARM_SUBSCRIPTION_ID
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_GCP credentials_json > ~/gcp-credentials.json
      || exit $?
    - export GOOGLE_APPLICATION_CREDENTIALS=~/gcp-credentials.json
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    script:
-   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7")
?                                                                                        -
+   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7"
+     --no-worktree)
    - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
      -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
      --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
    variables:
      TARGETS: ./tests/windows/domain-test
      TEAM: windows-agent

.new-e2e_windows_msi

  .new-e2e_windows_msi:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    artifacts:
      expire_in: 2 weeks
      paths:
      - $E2E_OUTPUT_DIR
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - WINDOWS_AGENT_VERSION=$(invoke agent.version) || exit $?; export WINDOWS_AGENT_VERSION
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_e2e.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_public_key_rsa > $E2E_PUBLIC_KEY_PATH
      || exit $?
    - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
      $AGENT_QA_E2E ssh_key_rsa > $E2E_PRIVATE_KEY_PATH || exit $?
    - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
    - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE client_id)
      || exit $?; export ARM_CLIENT_ID
    - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE token)
      || exit $?; export ARM_CLIENT_SECRET
    - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE tenant_id)
      || exit $?; export ARM_TENANT_ID
    - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE subscription_id)
      || exit $?; export ARM_SUBSCRIPTION_ID
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_GCP credentials_json > ~/gcp-credentials.json
      || exit $?
    - export GOOGLE_APPLICATION_CREDENTIALS=~/gcp-credentials.json
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_e2e_deps
    script:
-   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7")
?                                                                                        -
+   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7"
+     --no-worktree)
    - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
      -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
      --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
    stage: e2e
    tags:
    - arch:amd64
    variables:
      E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
      E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
      E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
      E2E_PIPELINE_ID: $CI_PIPELINE_ID
      E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
      E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      EXTRA_PARAMS: --run "$E2E_MSI_TEST$"
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      SHOULD_RUN_IN_FLAKES_FINDER: 'true'
      TARGETS: ./tests/windows/install-test
      TEAM: windows-agent

check_pkg_size-amd64-a7

  check_pkg_size-amd64-a7:
    before_script:
    - "declare -Ar max_sizes=(\n    [\"datadog-agent\"]=\"140000000\"\n    [\"datadog-iot-agent\"\
      ]=\"10000000\"\n    [\"datadog-dogstatsd\"]=\"10000000\"\n    [\"datadog-heroku-agent\"\
      ]=\"70000000\"\n)\n"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - agent_deb-x64-a7
    - iot_agent_deb-x64
    - dogstatsd_deb-x64
    - agent_heroku_deb-x64-a7
    - agent_rpm-x64-a7
    - iot_agent_rpm-x64
    - dogstatsd_rpm-x64
    - agent_suse-x64-a7
    - dogstatsd_suse-x64
    - iot_agent_suse-x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - ls -l $OMNIBUS_PACKAGE_DIR
    - if [[ "${ARCH}" == "amd64" ]]; then ls -l $OMNIBUS_PACKAGE_DIR_SUSE; fi
    - export failures=0
-   - export last_stable=$(inv release.get-release-json-value "last_stable::${MAJOR_VERSION}")
?                                                                                            -
+   - export last_stable=$(inv release.get-release-json-value "last_stable::${MAJOR_VERSION}"
+     --no-worktree)
    - "if [[ \"${ARCH}\" == \"amd64\" ]]; then ARCH_RPM_EXT=\"x86_64\"; else ARCH_RPM_EXT=\"\
      aarch64\"; fi\nfor flavor in ${FLAVORS}; do\n\n    if [[ \"${ARCH}\" == \"amd64\"\
      \ && \"$flavor\" != \"datadog-heroku-agent\" ]]; then\n      mkdir -p \"/tmp/stable/${flavor}/suse\"\
      \n      curl -sSL \"https://s3.amazonaws.com/yum.datadoghq.com/suse/stable/${MAJOR_VERSION}/${ARCH_RPM_EXT}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \ -o \"/tmp/stable/${flavor}/suse/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \n      set +e\n      inv package.compare-size --package-type \"${flavor} suse\
      \ rpm\" --last-stable \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\"\
      \ --new-package \"$OMNIBUS_PACKAGE_DIR_SUSE/${flavor}-${MAJOR_VERSION}.*.${ARCH_RPM_EXT}.rpm\"\
      \ --stable-package \"/tmp/stable/${flavor}/suse/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \n      failures=$((${failures}+$?))\n      set -e\n    fi\n\n    mkdir -p \"\
      /tmp/stable/${flavor}\"\n\n    curl -sSL \"https://s3.amazonaws.com/apt.datadoghq.com/pool/d/da/${flavor}_${last_stable}-1_${ARCH}.deb\"\
      \ -o \"/tmp/stable/${flavor}/${flavor}_${last_stable}-1_${ARCH}.deb\"\n\n    set\
      \ +e\n    inv package.compare-size --package-type \"${flavor} deb\" --last-stable\
      \ \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\" --new-package \"\
      $OMNIBUS_PACKAGE_DIR/${flavor}_${MAJOR_VERSION}*_${ARCH}.deb\" --stable-package\
      \ \"/tmp/stable/${flavor}/${flavor}_${last_stable}-1_${ARCH}.deb\"\n    failures=$((${failures}+$?))\n\
      \    set -e\n\n    if [[ \"$flavor\" != \"datadog-heroku-agent\" && ( \"${ARCH}\"\
      \ == \"amd64\" || \"$flavor\" != \"datadog-dogstatsd\") ]]; then\n      # We don't\
      \ build RPM packages for the heroku flavor\n      curl -sSL \"https://s3.amazonaws.com/yum.datadoghq.com/stable/${MAJOR_VERSION}/${ARCH_RPM_EXT}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \ -o \"/tmp/stable/${flavor}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\n\
      \      set +e\n      inv package.compare-size --package-type \"${flavor} rpm\"\
      \ --last-stable \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\" --new-package\
      \ \"$OMNIBUS_PACKAGE_DIR/${flavor}-${MAJOR_VERSION}.*.${ARCH_RPM_EXT}.rpm\" --stable-package\
      \ \"/tmp/stable/${flavor}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\n \
      \     failures=$((${failures}+$?))\n      set -e\n    fi\ndone\n"
    - if [ "${failures}" -ne "0" ]; then false; fi
    stage: pkg_metrics
    tags:
    - arch:amd64
    variables:
      ARCH: amd64
      FLAVORS: datadog-agent datadog-iot-agent datadog-dogstatsd datadog-heroku-agent
      MAJOR_VERSION: 7

check_pkg_size-arm64-a7

  check_pkg_size-arm64-a7:
    before_script:
    - "declare -Ar max_sizes=(\n    [\"datadog-agent\"]=\"140000000\"\n    [\"datadog-iot-agent\"\
      ]=\"10000000\"\n    [\"datadog-dogstatsd\"]=\"10000000\"\n)\n"
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - agent_deb-arm64-a7
    - iot_agent_deb-arm64
    - dogstatsd_deb-arm64
    - agent_rpm-arm64-a7
    - iot_agent_rpm-arm64
    rules:
    - if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
        == "true" || $DDR_WORKFLOW_ID != null
    script:
    - ls -l $OMNIBUS_PACKAGE_DIR
    - if [[ "${ARCH}" == "amd64" ]]; then ls -l $OMNIBUS_PACKAGE_DIR_SUSE; fi
    - export failures=0
-   - export last_stable=$(inv release.get-release-json-value "last_stable::${MAJOR_VERSION}")
?                                                                                            -
+   - export last_stable=$(inv release.get-release-json-value "last_stable::${MAJOR_VERSION}"
+     --no-worktree)
    - "if [[ \"${ARCH}\" == \"amd64\" ]]; then ARCH_RPM_EXT=\"x86_64\"; else ARCH_RPM_EXT=\"\
      aarch64\"; fi\nfor flavor in ${FLAVORS}; do\n\n    if [[ \"${ARCH}\" == \"amd64\"\
      \ && \"$flavor\" != \"datadog-heroku-agent\" ]]; then\n      mkdir -p \"/tmp/stable/${flavor}/suse\"\
      \n      curl -sSL \"https://s3.amazonaws.com/yum.datadoghq.com/suse/stable/${MAJOR_VERSION}/${ARCH_RPM_EXT}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \ -o \"/tmp/stable/${flavor}/suse/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \n      set +e\n      inv package.compare-size --package-type \"${flavor} suse\
      \ rpm\" --last-stable \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\"\
      \ --new-package \"$OMNIBUS_PACKAGE_DIR_SUSE/${flavor}-${MAJOR_VERSION}.*.${ARCH_RPM_EXT}.rpm\"\
      \ --stable-package \"/tmp/stable/${flavor}/suse/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \n      failures=$((${failures}+$?))\n      set -e\n    fi\n\n    mkdir -p \"\
      /tmp/stable/${flavor}\"\n\n    curl -sSL \"https://s3.amazonaws.com/apt.datadoghq.com/pool/d/da/${flavor}_${last_stable}-1_${ARCH}.deb\"\
      \ -o \"/tmp/stable/${flavor}/${flavor}_${last_stable}-1_${ARCH}.deb\"\n\n    set\
      \ +e\n    inv package.compare-size --package-type \"${flavor} deb\" --last-stable\
      \ \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\" --new-package \"\
      $OMNIBUS_PACKAGE_DIR/${flavor}_${MAJOR_VERSION}*_${ARCH}.deb\" --stable-package\
      \ \"/tmp/stable/${flavor}/${flavor}_${last_stable}-1_${ARCH}.deb\"\n    failures=$((${failures}+$?))\n\
      \    set -e\n\n    if [[ \"$flavor\" != \"datadog-heroku-agent\" && ( \"${ARCH}\"\
      \ == \"amd64\" || \"$flavor\" != \"datadog-dogstatsd\") ]]; then\n      # We don't\
      \ build RPM packages for the heroku flavor\n      curl -sSL \"https://s3.amazonaws.com/yum.datadoghq.com/stable/${MAJOR_VERSION}/${ARCH_RPM_EXT}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\
      \ -o \"/tmp/stable/${flavor}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\n\
      \      set +e\n      inv package.compare-size --package-type \"${flavor} rpm\"\
      \ --last-stable \"${last_stable}\" --threshold \"${max_sizes[${flavor}]}\" --new-package\
      \ \"$OMNIBUS_PACKAGE_DIR/${flavor}-${MAJOR_VERSION}.*.${ARCH_RPM_EXT}.rpm\" --stable-package\
      \ \"/tmp/stable/${flavor}/${flavor}-${last_stable}-1.${ARCH_RPM_EXT}.rpm\"\n \
      \     failures=$((${failures}+$?))\n      set -e\n    fi\ndone\n"
    - if [ "${failures}" -ne "0" ]; then false; fi
    stage: pkg_metrics
    tags:
    - arch:amd64
    variables:
      ARCH: arm64
      FLAVORS: datadog-agent datadog-iot-agent datadog-dogstatsd
      MAJOR_VERSION: 7

do-not-merge

  do-not-merge:
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs: []
    rules:
    - if: $CI_PIPELINE_SOURCE =~ /^schedule.*$/
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: always
    script:
    - "if [ ! -z \"$DATADOG_AGENT_BUILDIMAGES_SUFFIX\" ] ||\n   [ ! -z \"$DATADOG_AGENT_WINBUILDIMAGES_SUFFIX\"\
      \ ] ||\n   [ ! -z \"$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX\" ] ||\n   [ ! -z \"\
      $DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX\" ] ||\n   [ ! -z \"$DATADOG_AGENT_BTF_GEN_BUILDIMAGES_SUFFIX\"\
      \ ] ||\n   [ ! -z \"$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX\" ]; then\n  echo\
      \ \"Pull request uses non-empty BUILDIMAGES_SUFFIX variable\"\n  echo \"This workflow\
      \ fails so that the pull request cannot be merged\"\n  exit 1\nfi\n"
-   - inv -e release.check-omnibus-branches || exit 1
+   - inv -e release.check-omnibus-branches --no-worktree || exit 1
?                                           ++++++++++++++
    - exit 0
    stage: check_merge
    tags:
    - arch:amd64

new-e2e-windows-agent-domain-tests-a7-x86_64

  new-e2e-windows-agent-domain-tests-a7-x86_64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    artifacts:
      expire_in: 2 weeks
      paths:
      - $E2E_OUTPUT_DIR
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - WINDOWS_AGENT_VERSION=$(invoke agent.version) || exit $?; export WINDOWS_AGENT_VERSION
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_e2e.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_public_key_rsa > $E2E_PUBLIC_KEY_PATH
      || exit $?
    - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
      $AGENT_QA_E2E ssh_key_rsa > $E2E_PRIVATE_KEY_PATH || exit $?
    - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
    - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE client_id)
      || exit $?; export ARM_CLIENT_ID
    - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE token)
      || exit $?; export ARM_CLIENT_SECRET
    - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE tenant_id)
      || exit $?; export ARM_TENANT_ID
    - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE subscription_id)
      || exit $?; export ARM_SUBSCRIPTION_ID
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_GCP credentials_json > ~/gcp-credentials.json
      || exit $?
    - export GOOGLE_APPLICATION_CREDENTIALS=~/gcp-credentials.json
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_e2e_deps
    - deploy_windows_testing-a7
    rules:
    - if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null
    - if: $RUN_E2E_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_E2E_TESTS == "on"
      when: on_success
    - if: $CI_COMMIT_BRANCH == "main"
      when: on_success
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      when: on_success
    - if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
      when: on_success
    - changes:
        compare_to: main
        paths:
        - .gitlab/e2e/e2e.yml
        - test/new-e2e/pkg/**/*
        - test/new-e2e/go.mod
        - flakes.yaml
    - changes:
        compare_to: main
        paths:
        - tools/windows/DatadogAgentInstaller/**/*
        - .gitlab/e2e_install_packages/windows.yml
        - test/new-e2e/tests/windows/install-test/**/*
        - test/new-e2e/tests/windows/domain-test/**/*
        - tasks/msi.py
      when: on_success
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - allow_failure: true
      when: manual
    script:
-   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7")
?                                                                                        -
+   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7"
+     --no-worktree)
    - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
      -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
      --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
    stage: e2e_install_packages
    tags:
    - arch:amd64
    timeout: 1h15m
    variables:
      AGENT_MAJOR_VERSION: 7
      E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
      E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
      E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
      E2E_PIPELINE_ID: $CI_PIPELINE_ID
      E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
      E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      SHOULD_RUN_IN_FLAKES_FINDER: 'true'
      TARGETS: ./tests/windows/domain-test
      TEAM: windows-agent
      WINDOWS_AGENT_ARCH: x86_64

new-e2e-windows-agent-msi-upgrade-windows-server-a7-x86_64

  new-e2e-windows-agent-msi-upgrade-windows-server-a7-x86_64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    artifacts:
      expire_in: 2 weeks
      paths:
      - $E2E_OUTPUT_DIR
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - WINDOWS_AGENT_VERSION=$(invoke agent.version) || exit $?; export WINDOWS_AGENT_VERSION
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_e2e.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_public_key_rsa > $E2E_PUBLIC_KEY_PATH
      || exit $?
    - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
      $AGENT_QA_E2E ssh_key_rsa > $E2E_PRIVATE_KEY_PATH || exit $?
    - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
    - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE client_id)
      || exit $?; export ARM_CLIENT_ID
    - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE token)
      || exit $?; export ARM_CLIENT_SECRET
    - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE tenant_id)
      || exit $?; export ARM_TENANT_ID
    - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE subscription_id)
      || exit $?; export ARM_SUBSCRIPTION_ID
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_GCP credentials_json > ~/gcp-credentials.json
      || exit $?
    - export GOOGLE_APPLICATION_CREDENTIALS=~/gcp-credentials.json
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_e2e_deps
    - deploy_windows_testing-a7
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - changes:
        compare_to: main
        paths:
        - tools/windows/DatadogAgentInstaller/**/*
        - .gitlab/e2e_install_packages/windows.yml
        - test/new-e2e/tests/windows/install-test/**/*
        - test/new-e2e/tests/windows/domain-test/**/*
        - tasks/msi.py
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_E2E_TESTS == "off"
      when: never
    - if: ($CI_COMMIT_BRANCH == "main"  || $DEPLOY_AGENT == "true" || $RUN_E2E_TESTS
        == "on" || $DDR_WORKFLOW_ID != null) && $RUN_E2E_TESTS != "off"
    - if: $RUN_E2E_TESTS == "auto"
      variables:
        E2E_OSVERS: $E2E_BRANCH_OSVERS
    - if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null
      when: never
    script:
-   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7")
?                                                                                        -
+   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7"
+     --no-worktree)
    - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
      -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
      --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
    stage: e2e_install_packages
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
      E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
      E2E_MSI_TEST: TestUpgrade
      E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
      E2E_PIPELINE_ID: $CI_PIPELINE_ID
      E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
      E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      EXTRA_PARAMS: --run "$E2E_MSI_TEST$"
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      SHOULD_RUN_IN_FLAKES_FINDER: 'true'
      TARGETS: ./tests/windows/install-test
      TEAM: windows-agent
      WINDOWS_AGENT_ARCH: x86_64

new-e2e-windows-agent-msi-windows-server-a7-x86_64

  new-e2e-windows-agent-msi-windows-server-a7-x86_64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    artifacts:
      expire_in: 2 weeks
      paths:
      - $E2E_OUTPUT_DIR
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
      when: always
    before_script:
    - WINDOWS_AGENT_VERSION=$(invoke agent.version) || exit $?; export WINDOWS_AGENT_VERSION
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_e2e.tar.xz
    - mkdir -p ~/.aws
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config
      || exit $?
    - export AWS_PROFILE=agent-qa-ci
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_public_key_rsa > $E2E_PUBLIC_KEY_PATH
      || exit $?
    - touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
      $AGENT_QA_E2E ssh_key_rsa > $E2E_PRIVATE_KEY_PATH || exit $?
    - pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
    - ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE client_id)
      || exit $?; export ARM_CLIENT_ID
    - ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE token)
      || exit $?; export ARM_CLIENT_SECRET
    - ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE tenant_id)
      || exit $?; export ARM_TENANT_ID
    - ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_AZURE subscription_id)
      || exit $?; export ARM_SUBSCRIPTION_ID
    - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_GCP credentials_json > ~/gcp-credentials.json
      || exit $?
    - export GOOGLE_APPLICATION_CREDENTIALS=~/gcp-credentials.json
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
    needs:
    - go_e2e_deps
    - deploy_windows_testing-a7
    parallel:
      matrix:
      - E2E_MSI_TEST: TestInstall
      - E2E_MSI_TEST: TestRepair
      - E2E_MSI_TEST: TestUpgrade
      - E2E_MSI_TEST: TestUpgradeRollback
      - E2E_MSI_TEST: TestUpgradeRollbackWithoutCWS
      - E2E_MSI_TEST: TestUpgradeChangeUser
      - E2E_MSI_TEST: TestUpgradeFromV5
      - E2E_MSI_TEST: TestAgentUser/user_only
      - E2E_MSI_TEST: TestAgentUser/dotslash_user
      - E2E_MSI_TEST: TestAgentUser/hostname_user
      - E2E_MSI_TEST: TestAgentUser/LocalSystem
      - E2E_MSI_TEST: TestAgentUser/SYSTEM
      - E2E_MSI_TEST: TestInstallOpts
      - E2E_MSI_TEST: TestSubServicesOpts/all-subservices
      - E2E_MSI_TEST: TestSubServicesOpts/no-subservices
      - E2E_MSI_TEST: TestInstallAltDir
      - E2E_MSI_TEST: TestInstallExistingAltDir
      - E2E_MSI_TEST: TestInstallAltDirAndCorruptForUninstall
      - E2E_MSI_TEST: TestInstallFail
      - E2E_MSI_TEST: TestNPMUpgradeToNPM
      - E2E_MSI_TEST: TestNPMUpgradeNPMToNPM
      - E2E_MSI_TEST: TestNPMInstallWithAddLocal
      - E2E_MSI_TEST: TestNPMUpgradeFromBeta
      - E2E_MSI_TEST: TestUpgradeFromV6
    rules:
    - if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null
    - if: $RUN_E2E_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_E2E_TESTS == "on"
      when: on_success
    - if: $CI_COMMIT_BRANCH == "main"
      when: on_success
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      when: on_success
    - if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
      when: on_success
    - changes:
        compare_to: main
        paths:
        - .gitlab/e2e/e2e.yml
        - test/new-e2e/pkg/**/*
        - test/new-e2e/go.mod
        - flakes.yaml
    - changes:
        compare_to: main
        paths:
        - tools/windows/DatadogAgentInstaller/**/*
        - .gitlab/e2e_install_packages/windows.yml
        - test/new-e2e/tests/windows/install-test/**/*
        - test/new-e2e/tests/windows/domain-test/**/*
        - tasks/msi.py
      when: on_success
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - allow_failure: true
      when: manual
    script:
-   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7")
?                                                                                        -
+   - export LAST_STABLE_VERSION=$(invoke release.get-release-json-value "last_stable::7"
+     --no-worktree)
    - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
      -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
      --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
    stage: e2e_install_packages
    tags:
    - arch:amd64
    timeout: 1h15m
    variables:
      AGENT_MAJOR_VERSION: 7
      E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
      E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
      E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
      E2E_PIPELINE_ID: $CI_PIPELINE_ID
      E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
      E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      EXTRA_PARAMS: --run "$E2E_MSI_TEST$"
      KUBERNETES_CPU_REQUEST: 6
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 12Gi
      SHOULD_RUN_IN_FLAKES_FINDER: 'true'
      TARGETS: ./tests/windows/install-test
      TEAM: windows-agent
      WINDOWS_AGENT_ARCH: x86_64

single-machine-performance-regression_detector

  single-machine-performance-regression_detector:
    allow_failure: false
    artifacts:
      expire_in: 1 weeks
      paths:
      - submission_metadata
      - ${CI_COMMIT_SHA}-baseline_sha
      - outputs/report.md
      - outputs/regression_signal.json
      - outputs/bounds_check_signal.json
      - outputs/junit.xml
      - outputs/report.json
      - outputs/decision_record.md
      when: always
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - artifacts: false
      job: single_machine_performance-amd64-a7
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      when: never
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DATADOG_API_KEY
    - datadog-ci tag --level job --tags smp_failure_mode:"unknown"
    - mkdir outputs
    - git fetch origin
-   - SMP_BASE_BRANCH=$(inv release.get-release-json-value base_branch)
+   - SMP_BASE_BRANCH=$(inv release.get-release-json-value base_branch --no-worktree)
?                                                                     ++++++++++++++
    - echo "Looking for merge base for branch ${SMP_BASE_BRANCH}"
    - SMP_MERGE_BASE=$(git merge-base ${CI_COMMIT_SHA} origin/${SMP_BASE_BRANCH})
    - echo "Merge base is ${SMP_MERGE_BASE}"
    - AWS_NAMED_PROFILE="single-machine-performance"
    - SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)
      || exit $?
    - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
    - SMP_AGENT_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT agent_team_id)
      || exit $?
    - SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT api_url) || exit
      $?
    - SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT bot_login)
      || exit $?
    - SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT bot_token)
      || exit $?
    - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
    - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
    - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
    - aws --profile single-machine-performance s3 cp s3://smp-cli-releases/v${SMP_VERSION}/x86_64-unknown-linux-gnu/smp
      smp
    - chmod +x smp
    - BASELINE_SHA="${SMP_MERGE_BASE}"
    - echo "Computing baseline..."
    - echo "Checking if commit ${BASELINE_SHA} is recent enough..."
    - FOUR_DAYS_BEFORE_NOW=$(date --date="-4 days +1 hour" "+%s")
    - BASELINE_COMMIT_TIME=$(git -c log.showSignature=false show --no-patch --format=%ct
      ${BASELINE_SHA})
    - "if [[ ${BASELINE_COMMIT_TIME} -le ${FOUR_DAYS_BEFORE_NOW} ]]\nthen\n    echo\
      \ \"ERROR: Merge-base of this branch is too old for SMP. Please update your branch\
      \ by merging an up-to-date main branch into your branch or by rebasing it on an\
      \ up-to-date main branch.\"\n    datadog-ci tag --level job --tags smp_failure_mode:\"\
      merge-base-too-old\"\n    exit 1\nfi\n"
    - echo "Commit ${BASELINE_SHA} is recent enough"
    - echo "Checking if image exists for commit ${BASELINE_SHA}..."
    - "while [[ ! $(aws ecr describe-images --region us-west-2 --profile single-machine-performance\
      \ --registry-id \"${SMP_ACCOUNT_ID}\" --repository-name \"${SMP_AGENT_TEAM_ID}-agent\"\
      \ --image-ids imageTag=\"${BASELINE_SHA}-7-amd64\") ]]\ndo\n    echo \"No image\
      \ exists for ${BASELINE_SHA} - checking predecessor of ${BASELINE_SHA} next\"\n\
      \    BASELINE_SHA=$(git rev-parse ${BASELINE_SHA}^)\n    echo \"Checking if commit\
      \ ${BASELINE_SHA} is recent enough...\"\n    BASELINE_COMMIT_TIME=$(git -c log.showSignature=false\
      \ show --no-patch --format=%ct ${BASELINE_SHA})\n    if [[ ${BASELINE_COMMIT_TIME}\
      \ -le ${FOUR_DAYS_BEFORE_NOW} ]]\n    then\n        echo \"ERROR: Merge-base of\
      \ this branch is too old for SMP. Please update your branch by merging an up-to-date\
      \ main branch into your branch or by rebasing it on an up-to-date main branch.\"\
      \n        datadog-ci tag --level job --tags smp_failure_mode:\"merge-base-too-old-predecessor\"\
      \n        exit 1\n    fi\n    echo \"Commit ${BASELINE_SHA} is recent enough\"\
      \n    echo \"Checking if image exists for commit ${BASELINE_SHA}...\"\ndone\n"
    - echo "Image exists for commit ${BASELINE_SHA}"
    - echo "Baseline SHA is ${BASELINE_SHA}"
    - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
    - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
      "s3://${SMP_AGENT_TEAM_ID}-smp-artifacts/information/"
    - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_AGENT_TEAM_ID}-agent:${BASELINE_SHA}-7-amd64
    - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
    - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_AGENT_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-amd64
    - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
    - SMP_TAGS="ci_pipeline_id=${CI_PIPELINE_ID},ci_job_id=${CI_JOB_ID}"
    - echo "Tags passed through SMP are ${SMP_TAGS}"
    - RUST_LOG="info,aws_config::profile::credentials=error"
    - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_AGENT_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
      \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
      \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
      \ test/regression/ \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
      \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
      \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
      \ $exit_code\n}\n"
    - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
    - echo "SMP Job Id is ${SMP_JOB_ID}"
    - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_AGENT_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
      \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
      \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
      \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_AGENT_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
      \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
      smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
      \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
    - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
    - datadog-ci junit upload --service datadog-agent outputs/junit.xml
    - datadog-ci tag --level job --tags smp_failure_mode:"none"
    - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_AGENT_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
      \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
      \ job --tags smp_optimization_goal:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_bounds_check:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_AGENT_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
      \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
      \ job --tags smp_bounds_check:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"failed"
    - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
      \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
      Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
      \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
      \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
      \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
      \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
      \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
      \ {})\n            comparison = results.get('comparison', [])\n            num_total\
      \ = len(comparison)\n            failed_replicates = [\n                replicate\
      \ for replicate in comparison if not replicate.get('passed', False)\n        \
      \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
      \ = num_total - num_failed\n            if failed_replicates:\n              \
      \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
      \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
      \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
      \ = True\n            else:\n                decision_record.append(\n       \
      \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
      \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
      \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
      \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
      \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
      \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
      n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
      Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
      Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"passed"
    stage: functional_test
    tags:
    - arch:amd64
    timeout: 1h10m
    variables:
      SMP_VERSION: 0.19.3

Changes Summary

Removed	Modified	Added	Renamed
0	11	0	0

ℹ️ Diff available in the job log.

…sks called within jobs

chouetz

One modification request on the re matching command, otherwise only minor comments. Thanks!

tasks/libs/ciproviders/github_api.py

tasks/libs/releasing/notes.py

tasks/release.py

CelianR · 2024-12-06T09:19:43Z

/merge

dd-devflow · 2024-12-06T09:19:52Z

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-12-06 09:19:52 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2024-12-06 10:19:21 UTC ℹ️ MergeQueue: merge request added to the queue

The median merge time in main is 23m.

The workflow fails (https://github.com/DataDog/datadog-agent/actions/runs/12231431220/job/34114505590) because the new input option brought by #30062

CelianR added 4 commits October 11, 2024 16:56

[a6-releasing] Added agent context context manager

3eb28f0

[a6-releasing] Doc

8868731

[a6-releasing] Default modules

f6797e7

[a6-releasing] Dynamic default module loading magic + test [skip ci]

3ee5297

CelianR added changelog/no-changelog qa/no-code-change No code change in Agent code requiring validation team/agent-devx-infra labels Oct 11, 2024

CelianR self-assigned this Oct 11, 2024

CelianR changed the title ~~Agent 6 release tasks support~~ [ACIX-453] Agent 6 release tasks support Oct 11, 2024

CelianR added 21 commits October 14, 2024 09:59

[a6-releasing] Cleaned agent context code

4ae2b29

[a6-releasing] Agent6 set context

c4afe7a

[a6-releasing] Update modules

7b5331a

[a6-releasing] Updated message

a3a647e

[a6-releasing] Updated tag_* [skip ci]

c616dbc

[a6-releasing] Formatted doc

ebf1512

[a6-releasing] notes: Updated changelog notes

be93145

[a6-releasing] Updated finish

9f00afb

[a6-releasing] Updated backport branch name [skip ci]

cd73e2c

[a6-releasing] Updated agent_context to clean code

82e7f1f

[skip ci] Fixed info message

7abffde

[a6-releasing] Fixed agent context defaults

c6fd47e

[a6-releasing] tmp

d62e017

[a6-releasing] Fixed defaults

a4e8c72

[a6-releasing] tmp

18ddc15

[a6-releasing] Removed tests

7c781e2

[a6-releasing] tmp

7e11dc2

[a6-releasing] tmp

28aa4f8

[a6-releasing] tmp

89e9a57

[a6-releasing] tmp

8e6bf10

[a6-releasing] tmp

87ace4c

CelianR added 2 commits November 27, 2024 13:40

[a6-releasing] Small fixes

42102ea

[a6-releasing] release: Added --no-worktree to get / set json

13f5380

CelianR requested review from a team as code owners November 27, 2024 12:55

pducolin approved these changes Nov 27, 2024

View reviewed changes

CelianR added 3 commits November 27, 2024 14:52

[a6-releasing] Disabled worktree / added task argument for release ta…

b73255e

…sks called within jobs

[a6-releasing] [merge]

a885616

[a6-releasing] do-not-merge: Fixed

67e3173

scottopell approved these changes Nov 27, 2024

View reviewed changes

gjulianm approved these changes Nov 27, 2024

View reviewed changes

chouetz reviewed Nov 30, 2024

View reviewed changes

CelianR added 5 commits December 5, 2024 17:16

[a6-releasing] suggestions: startswith 6.53

25a2aa6

[a6-releasing] suggestions: yes -> trust

d70e2d7

[a6-releasing] suggestions: Removed "agent 7 version"

5f437ff

[a6-releasing] suggestions: Removed "agent 7 version"

bcf2aa0

[a6-releasing] [merge]

67fe4e7

chouetz approved these changes Dec 5, 2024

View reviewed changes

[a6-releasing] Fixed regex

798a92c

iliakur approved these changes Dec 6, 2024

View reviewed changes

[a6-releasing] [merge]

abb177c

dd-mergequeue bot merged commit f7625c5 into main Dec 6, 2024
316 of 334 checks passed

dd-mergequeue bot deleted the celian/a6-releasing branch December 6, 2024 10:48

github-actions bot added this to the 7.62.0 milestone Dec 6, 2024

FlorentClarret mentioned this pull request Dec 9, 2024

Fix the Create RC PR Workflow #31892

Merged

chouetz added a commit that referenced this pull request Dec 11, 2024

fix(create_rc): Pass the option to the invoke task

539b3f2

The workflow fails (https://github.com/DataDog/datadog-agent/actions/runs/12231431220/job/34114505590) because the new input option brought by #30062

chouetz mentioned this pull request Dec 11, 2024

fix(create_rc): Pass the option to the invoke task #32054

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ACIX-453] Agent 6 release tasks support #30062

[ACIX-453] Agent 6 release tasks support #30062

CelianR commented Oct 11, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Nov 27, 2024 •

edited

Loading

Modified Jobs

chouetz left a comment

CelianR commented Dec 6, 2024

dd-devflow bot commented Dec 6, 2024 •

edited

Loading

[ACIX-453] Agent 6 release tasks support #30062

[ACIX-453] Agent 6 release tasks support #30062

Conversation

CelianR commented Oct 11, 2024 • edited Loading

What does this PR do?

Workflow

Motivation

Describe how to test/QA your changes

Possible Drawbacks / Trade-offs

Additional Notes

agent-platform-auto-pr bot commented Nov 27, 2024 • edited Loading

Gitlab CI Configuration Changes

Modified Jobs

Changes Summary

chouetz left a comment

Choose a reason for hiding this comment

CelianR commented Dec 6, 2024

dd-devflow bot commented Dec 6, 2024 • edited Loading

Devflow running: /merge

CelianR commented Oct 11, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Nov 27, 2024 •

edited

Loading

dd-devflow bot commented Dec 6, 2024 •

edited

Loading

Devflow running: `/merge`