Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gitlab] Migrate docker publish jobs to k8s runners #17270

Merged
merged 3 commits into from
Jun 14, 2023
Merged

[gitlab] Migrate docker publish jobs to k8s runners #17270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e40954d
[gitlab] Migrate docker publish jobs to k8s runners
KSerrania May 23, 2023
8cbfa27
Also cover jobs in the deploy stages
KSerrania May 24, 2023
2345b61
Merge branch 'main' into kserrania/migrate-docker-publish-job-k8s
KSerrania Jun 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitlab/deploy_6/container.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
stage: deploy6
dependencies: []
before_script:
- source /root/.bashrc
- if [[ "$VERSION" == "" ]]; then export VERSION="$(inv agent.version --major-version 6 --url-safe)"; fi
- export IMG_SOURCES="${SRC_AGENT}:v${CI_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-6${JMX}-amd64,${SRC_AGENT}:v${CI_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-6${JMX}-arm64"
- export IMG_DESTINATIONS="${AGENT_REPOSITORY}:${VERSION}${JMX}"
Expand Down
2 changes: 2 additions & 0 deletions .gitlab/deploy_7/container.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
stage: deploy7
dependencies: []
before_script:
- source /root/.bashrc
- if [[ "$VERSION" == "" ]]; then export VERSION="$(inv agent.version --major-version 7 --url-safe)"; fi
- export IMG_BASE_SRC="${SRC_AGENT}:v${CI_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}"
- export IMG_LINUX_SOURCES="${IMG_BASE_SRC}-7${JMX}-amd64,${IMG_BASE_SRC}-7${JMX}-arm64"
Expand Down Expand Up @@ -55,6 +56,7 @@ deploy_containers-dogstatsd:
!reference [.on_deploy_a7_manual_auto_on_rc]
dependencies: []
before_script:
- source /root/.bashrc
- export VERSION="$(inv agent.version --major-version 7 --url-safe)"
- export IMG_SOURCES="${SRC_DSD}:v${CI_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-amd64,${SRC_DSD}:v${CI_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-arm64"
- export IMG_DESTINATIONS="${DSD_REPOSITORY}:${VERSION}"
Expand Down
1 change: 1 addition & 0 deletions .gitlab/deploy_dca.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
stage: deploy_dca
dependencies: []
before_script:
- source /root/.bashrc
- if [[ "$VERSION" == "" ]]; then export VERSION="$(inv agent.version --major-version 7 --url-safe)"; fi
- if [[ "$CLUSTER_AGENT_REPOSITORY" == "" ]]; then export CLUSTER_AGENT_REPOSITORY="cluster-agent"; fi
- export IMG_BASE_SRC="${SRC_DCA}:v${CI_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}"
Expand Down
3 changes: 2 additions & 1 deletion .gitlab/docker_common/publish_job_templates.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,13 @@

.docker_publish_job_definition:
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
tags: ["runner:main"]
tags: ["arch:amd64"]
variables:
<<: *docker_variables
IMG_VARIABLES: ""
IMG_SIGNING: ""
script: # We can't use the 'trigger' keyword on manual jobs, otherwise they can't be run if the pipeline fails and is retried
- source /root/.bashrc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anything magic in there? I don't think we have it on other jobs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically the same reason as in #17179: source /root/.bashrc is run in the entrypoint of the image, but in k8s runners, entrypoints are overwritten, so we need to run this manually.

In the case of this job, source /root/.bashrc runs conda activate ddpy3, which is necessary to get the aws CLI in the PATH.

In the long run, we want to get rid of this line, but there's some more work to do in the build images to get to this.

- export GITLAB_TOKEN=$(aws ssm get-parameter --region us-east-1 --name ci.datadog-agent.gitlab_pipelines_scheduler_token --with-decryption --query "Parameter.Value" --out text)
- ECR_RELEASE_SUFFIX="${CI_COMMIT_TAG+-release}"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA})#\1${ECR_RELEASE_SUFFIX}#g" <<<"$IMG_VARIABLES")"
Expand Down