Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The image-syncer documentation update #11948

Merged
merged 5 commits into from
Sep 24, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 0 additions & 28 deletions cmd/image-syncer/Makefile

This file was deleted.

163 changes: 121 additions & 42 deletions cmd/image-syncer/README.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,97 @@
# image-syncer

Contents:

- [Overview](#overview)
- [User Guide](#user-guide)
- [Workflow](#workflow)
- [Example Workflow](#example-workflow)
- [Example Images List File](#example-images-list-file)
- [Developer Guide](#developer-guide)
- [Syncing Process](#syncing-process)
- [Reusable Workflow](#reusable-workflow)
- [Image Syncer Container Image](#image-syncer-container-image)
- [Flags](#flags)
- [Environment Variables](#environment-variables)
- [Infrastucture as Code](#infrastucture-as-code)
dekiel marked this conversation as resolved.
Show resolved Hide resolved

## Overview

image-syncer is used to copy container images from one registry to another.
The main use case is to preserve images from third party registries in your own registry that you can rely on.
The image-syncer is used to copy container images between two registries.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
It copies images **only when they are not present** in the target repo.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
The tool guarantees that **tags are immutable** in the target repo.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
That means that if the image tag is already present in the target registry, it will not be overwritten by the new image.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
The tool can be used in a GitHub workflow to synchronize images defined in a YAML file maintained in the repository.

## User Guide

The developers can use image-syncer to synchronize images defined in the YAML file maintained in their repository.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
Each team defines the list of images they want to synchronize.

Follow the steps below to use the image-syncer for your repository:
dekiel marked this conversation as resolved.
Show resolved Hide resolved

1. Create a workflow that calls
the [image-syncer reusable workflow](https://github.com/kyma-project/test-infra/blob/main/.github/workflows/image-syncer.yml).
See the example [workflow](#example-workflow) below.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
2. Create a PR with the YAML file with the list of images you want to synchronize.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
See the example [images list file](#example-images-list-file) below.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
3. Merge the PR to synchronize the images.

### Workflow

It copies images **only when they are not present** in the target repo. That guarantees that **tags are immutable**.
> [!IMPORTANT]
> The image-syncer running for pull request will always run in dry-run mode.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
> The workflow calling image-syncer reusable workflow on pull request must use the pull_request_target event as a trigger.
dekiel marked this conversation as resolved.
Show resolved Hide resolved

Syncing process steps:
1. Pull an image from source.
2. Check if the image name contains the multi-platform SHA256 digest. To get this digest, run `docker run mplatform/manifest-tool inspect {your-image}:{image-version}` and copy the first digest this command returns. For the non-multi-platform images supporting only amd64 architecture, use their amd64 digest and add the `amd64only: true` parameter.
3. If the image name contains digest, re-tag the target image with the tag instead of SHA256 digest.
4. Check if the image exists in target.
5. If the image does not exist, re-tag the image and push to the target registry.
If the image exists, compare the IDs. If they are different, synchronization is interrupted. It means that something is wrong and the source image was changed (this should not happen).
6. Push the signature to the target repository.
See
supported [inputs](https://github.com/kyma-project/test-infra/blob/4df11c5384a5c7ac3ce76b726e17dee6aba07f79/.github/workflows/image-syncer.yml#L5)
by the image-syncer reusable workflow.
dekiel marked this conversation as resolved.
Show resolved Hide resolved

These steps guarantee that images in your registry are immutable and verified.
The image-syncer supports following GitHub events:
dekiel marked this conversation as resolved.
Show resolved Hide resolved

## Usage
- pull_request_target
- push

To run image-syncer, use:
```bash
go run main.go \
--images-file={PATH_TO_A_YAML_FILE_CONTAINING_SYNC_DEFINITION} \
--target-repo-auth-key={PATH_TO_A_JSON_KEY_FILE} \
--dry-run=true
The workflow must call the image-syncer reusable workflow from the main branch of the test-infra repository.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
The value of uses must be `kyma-project/test-infra/.github/workflows/image-syncer.yml@main`.

#### Example Workflow

```yaml
name: pull-sync-external-images

on:
pull_request_target:
branches:
- main
types: [ opened, edited, synchronize, reopened, ready_for_review ]
paths:
- "external-images.yaml"

permissions:
id-token: write # This is required for requesting the JWT token
contents: read # This is required for actions/checkout

jobs:
sync-external-images:
uses: kyma-project/test-infra/.github/workflows/image-syncer.yml@main
with:
debug: true
```
### Definition File

As an input parameter, image-syncer takes a file having the following structure:
### Example Images List File

> [!IMPORTANT]
> The image-syncer expects the file external-images.yaml with the list of images in root directory of the repository.
dekiel marked this conversation as resolved.
Show resolved Hide resolved

As an input parameter, image-syncer takes a file having the following structure:

- source: the source image to be copied. It can be an image name with a tag or a digest.
- tag: the tag of the image in the target repository. This is required when the source uses digest to identify the image.
- amd64Only: a boolean value that indicates if it's allowed to synchronize the image manifest instead of image index.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
This is required when the source image is not a multi-platform image.

```yaml
targetRepoPrefix: "eu.gcr.io/kyma-project/external/"
images:
- source: "bitnami/keycloak-gatekeeper:9.0.3"
- source: "bitnami/postgres-exporter:0.8.0-debian-10-r28"
Expand All @@ -45,28 +104,48 @@ images:
amd64Only: true
```

### Flags
## Developer Guide

```
Usage:
image-syncer [flags]

Flags:
--debug Enables the debug mode [SYNCER_DEBUG]
--dry-run Enables the dry-run mode [SYNCER_DRY_RUN]
-h, --help help for image-syncer
-i, --images-file string Specifies the path to the YAML file that contains list of images [SYNCER_IMAGES_FILE]
-t, --target-repo-auth-key string Specifies the JSON key file used for authorization to the target repository [SYNCER_TARGET_REPO_AUTH_KEY]
```
The image-syncer tool consists of the following components:

- The image-syncer binary, written in Go. The binary is released as a container image.
- The image-syncer reusable workflow that can be called from the GitHub workflow.
- The image-syncer infrastructure resources are used to access the target registry.
dekiel marked this conversation as resolved.
Show resolved Hide resolved

### Image Syncer Binary

The image-syncer binary is responsible for synchronizing images between two registries.
The binary is released as a container image.
The [Dockerfile](https://github.com/kyma-project/test-infra/blob/main/cmd/image-syncer/Dockerfile) for the image-syncer binary.
dekiel marked this conversation as resolved.
Show resolved Hide resolved

> [!IMPORTANT]
> The amd64Only field in the images list file allows syncing the image manifest instead of the image index.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
> The flag does not prevent syncing other platforms.

#### Flags

The image-syncer binary accepts flags to configure the synchronization process.
Supported [flags](https://github.com/kyma-project/test-infra/blob/1df13d56ad523ce434e33284bb7e392ff897cd1b/cmd/image-syncer/main.go#L274-L282).
dekiel marked this conversation as resolved.
Show resolved Hide resolved

#### Environment Variables

All the flags can also be set using environment variables.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
The environment variables must be prefixed with `SYNCER` and use uppercase letters.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
Dash characters must be replaced with underscores.
For example, the `--dry-run` flag can be set using the `SYNCER_DRY_RUN` environment variable.
dekiel marked this conversation as resolved.
Show resolved Hide resolved

### Reusable Workflow

The image-syncer reusable workflow provides an execution environment for the image-syncer binary.
The workflow is called from the GitHub workflow defined in other repositories.
It authenticates in Google Cloud and provides the image-syncer binary with the required credentials.

### Environment Variables
### Infrastucture as Code

All the flags can also be set using these environment variables:
The image-syncer needs certain infrastructure resources to access the target registry.
dekiel marked this conversation as resolved.
Show resolved Hide resolved
The resources are defined in
the [infrastructure-as-code](https://github.com/kyma-project/test-infra/tree/main/configs/terraform/environments/prod) configuration.
dekiel marked this conversation as resolved.
Show resolved Hide resolved

| Name | Required | Description |
| :----------------------------- | :------: | :-------------------------------------------------------------------- |
| **SYNCER_IMAGES_FILE** | Yes | Path to the YAML file with the sync definition, provided as a string.|
| **SYNCER_TARGET_REPO_AUTH_KEY**| Yes | Path to the JSON key file, provided as a string.|
| **SYNCER_DRY_RUN** | No | Value controlling the `dry run` mode, provided as a boolean.|
|**SYNCER_DEBUG**| No | Variable that enables the `debug` mode.|
- The image-syncer [resources](https://github.com/kyma-project/test-infra/blob/main/configs/terraform/environments/prod/image-syncer.tf).
- The
image-syncer [variables](https://github.com/kyma-project/test-infra/blob/main/configs/terraform/environments/prod/image-syncer-variables.tf).