Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(publish) abstract environment variables and introduce feature flags #784

Conversation

dduportal
Copy link
Contributor

@dduportal dduportal commented May 29, 2024

While working on #777 for the Update Center migration to a mirror system, we realized the following problems with the current publish.sh script:

  • It's hard to test a change until it's in production => we would want to enable/disable some steps to only focus testing on others in order to have enough confidence to merge PRs without breaking the production (neither requiring a fully fledged staging environment)
  • There are parameters written in the script itself which requires SREs to open an (untestable for today) PR and merge it as soon as possible when rotating expired credentials/changing configurations of bucket, etc.

This PR is a proposal to solve these issues with:

  • Feature flagging for the top-levels stages of this script, through en environment variable which has a default value (same behavior as today) and can be overridden by the caller
  • Definition of the Update Center sync. tasks through an environment variable (same as above: default value which can be overridden) and a convention that each "sync" task will load their own environment file to get their parameters and credentials

Example of the expected source file structures to be set in a ZIP credential file:

####### File env-rsync-pkg
export UPDATES_SITE=updates.jenkins.io
export RSYNC_USER=mirrorbrain
export RSYNC_GROUP=www-data
export RSYNC_REMOTE_DIR=/var/www/updates.jenkins.io
export FILESHARE_SYNC_SOURCE=./www2/

####### File env-azsync-content
export STORAGE_NAME=updatesjenkinsio
export STORAGE_FILESHARE=updates-jenkins-io
export STORAGE_DURATION_IN_MINUTE=5
export STORAGE_PERMISSIONS=dlrw
export JENKINS_INFRA_FILESHARE_CLIENT_ID='<redacted>'
export JENKINS_INFRA_FILESHARE_CLIENT_SECRET='<redacted>'
export JENKINS_INFRA_FILESHARE_TENANT_ID='<redacted>'
export FILESHARE_SYNC_SOURCE=./www-content/

####### File env-s3sync-westeurope
export AWS_ACCESS_KEY_ID='<redacted>'
export AWS_SECRET_ACCESS_KEY='<redacted>'
export AWS_DEFAULT_REGION='auto'
export BUCKET_NAME=westeurope-updates-jenkins-io
export BUCKET_ENDPOINT_URL=https://8d1838a43923148c5cee18ccc356a594.r2.cloudflarestorage.com
export FILESHARE_SYNC_SOURCE=./www-content/

Note: This PR is ready to review but not to merge as the env. files need to be generated and set up AND no test done (yet) on real life

Signed-off-by: Damien Duportal <[email protected]>
site/publish.sh Outdated Show resolved Hide resolved
Co-authored-by: Stéphane MERLE <[email protected]>
dduportal added a commit to jenkins-infra/azure that referenced this pull request May 30, 2024
Related to jenkins-infra/helpdesk#2649 and
blocking jenkins-infra/update-center2#784 and
jenkins-infra/charts-secrets#30

This PR simplifies and rename outputs to ensure the script used to
generate the Update Center credential ZIP file in
jenkins-infra/charts-secrets#30 can extract the
proper output with the naming convention from
jenkins-infra/update-center2#777 and
jenkins-infra/update-center2#784 (`www-content`
and `www-redirections`).

- It avoids multiple levels of escaping challenges (current output
requires 3 levels of escaping: in HCL syntax AND in shell syntax for the
initial call AND on shell syntax in the file template itself)
- The only "coupling" in Terraform HCL is the output name which is an
acceptable contract: it avoid having to change things in the terraform
state when only renaming a variable
- No more risk to have a local env file committed by error and exposing
the secrets

---------

Signed-off-by: Damien Duportal <[email protected]>
@dduportal dduportal changed the base branch from master to chore/publish/abstract-env May 31, 2024 14:44
@dduportal dduportal marked this pull request as ready for review May 31, 2024 14:44
@dduportal
Copy link
Contributor Author

dduportal commented May 31, 2024

We're merging this PR into a specific branch to allow testing. We'll open a subsequent PR for the final review to master branch. Any code review should happen in #787

@dduportal dduportal merged commit bfb7db4 into jenkins-infra:chore/publish/abstract-env May 31, 2024
0 of 2 checks passed
@dduportal dduportal deleted the chore/publish/abstract-env branch May 31, 2024 15:00
dduportal added a commit that referenced this pull request Jun 4, 2024
…lags (#784)

* chore(publish) abstract environment variables and introduce feature flags

Signed-off-by: Damien Duportal <[email protected]>

* comments fixup

Signed-off-by: Damien Duportal <[email protected]>

* Update site/publish.sh

Co-authored-by: Stéphane MERLE <[email protected]>

* enhance rsync ssh and rename pkg sync-task

---------

Signed-off-by: Damien Duportal <[email protected]>
Co-authored-by: Stéphane MERLE <[email protected]>
dduportal added a commit that referenced this pull request Jul 8, 2024
…lag (#787)

* chore(publish) abstract environment variables and introduce feature flags (#784)

* chore(publish) abstract environment variables and introduce feature flags

Signed-off-by: Damien Duportal <[email protected]>

* comments fixup

Signed-off-by: Damien Duportal <[email protected]>

* Update site/publish.sh

Co-authored-by: Stéphane MERLE <[email protected]>

* enhance rsync ssh and rename pkg sync-task

---------

Signed-off-by: Damien Duportal <[email protected]>
Co-authored-by: Stéphane MERLE <[email protected]>

* Update site/publish.sh

* Apply suggestions from code review

Co-authored-by: Tim Jacomb <[email protected]>

---------

Signed-off-by: Damien Duportal <[email protected]>
Co-authored-by: Stéphane MERLE <[email protected]>
Co-authored-by: Tim Jacomb <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants