Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move all actual Agones releases images to GAR #2875

Merged
merged 8 commits into from
Dec 21, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions build/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -26,18 +26,14 @@
# base version target. This is usually the next release.
base_version = 1.29.0

# agones image release registry
release_registry = gcr.io/agones-images

#
# All of the following can be overwritten with environment variables
# or passed through directly when invoking the relevent Make targets
#

# Version defaults to the short hash of the latest commit
VERSION ?= $(base_version)-$(shell git rev-parse --short=7 HEAD)
# The registry that is being used to store docker images
REGISTRY ?= $(release_registry)

# kubectl configuration to use
KUBECONFIG ?= ~/.kube/config
# The (gcloud) test cluster that is being worked against
Expand Down
18 changes: 11 additions & 7 deletions build/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,10 +225,14 @@ Assuming that all works, let's exit the shell by typing `exit` and hitting enter
installing Agones next.

To prepare building and pushing images, let's set the REGISTRY environment variable to point to our new project.
You can [choose any registry region](https://cloud.google.com/container-registry/docs/pushing-and-pulling#choosing_a_registry_name)
but for this example, we'll just use `gcr.io`.

In your shell, run `export REGISTRY=gcr.io/<YOUR-PROJECT-ID>` which will overwrite the default registry settings in our
You can choose either Google Container Registry or Google Artifact Registry but you must set it explicitly.
For this guidance, we will use Google Artifact Registry.
You can [choose any registry region](https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling)
but for this example, we'll just use `us-docker.pkg.dev`.
Please follow the [instructions](https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling#before_you_begin) to create the registry
in your project properly before you contine.

In your shell, run `export REGISTRY=us-docker.pkg.dev/<YOUR-PROJECT-ID>/<YOUR-REGISTRY-NAME>` which will set the required `REGISTRY` parameters in our
Make targets. Then, to rebuild our images for this registry, we run `make build-images` again.

Before we can push the images, there is one more small step! So that we can run regular `docker push` commands
Expand Down Expand Up @@ -347,7 +351,7 @@ Prerequisites:
To begin, you need to set up the following environment variables:
- `KUBECONFIG` should point to the kubeconfig file used to access the cluster;
if unset, it defaults to `~/.kube/config`
- `REGISTRY` should point to your image repository of your choice (i.e. gcr.io/<YOUR-PROJECT-ID>)
- `REGISTRY` should point to your image repository of your choice (i.e. us-docker.pkg.dev/<YOUR-PROJECT-ID>/<YOUR-REGISTRY-NAME>)
- `IMAGE_PULL_SECRET` must contain the name of the secret required to pull the Agones images,
in case you're using a custom repository; if unset, no pull secret will be used
- `IMAGE_PULL_SECRET_FILE` must be initialized to the full path of the file containing
Expand Down Expand Up @@ -417,7 +421,7 @@ command from within `make shell` or locally, to refresh your authentication toke
The version of this build. Version defaults to the short hash of the latest commit.

### REGISTRY
The registry that is being used to store docker images. Defaults to gcr.io/agones-images - the release + CI registry.
The registry that is being used to store docker images. It doesn't have default value and has to be set explicitly.

### KUBECONFIG
The Kubernetes config file used to access the cluster. Defaults to `~/.kube/config` - the file used by default by kubectl.
Expand Down Expand Up @@ -725,7 +729,7 @@ Change AGONES_VERSION to a specific version you want to install.

#### `make gcloud-terraform-install`
Create GKE cluster and install current version of agones.
The current version should be built and pushed to release_registry beforehand:
The current version should be built and pushed to `$(REGISTRY)` beforehand:
```
make build-images
make push
Expand Down
4 changes: 2 additions & 2 deletions build/includes/google-cloud.mk
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,9 @@ gcloud-auth-cluster: $(ensure-build-image)
docker run --rm $(common_mounts) $(build_tag) gcloud container clusters get-credentials $(GCP_CLUSTER_NAME) --zone $(GCP_CLUSTER_LOCATION)

# authenticate our docker configuration so that you can do a docker push directly
# to the gcr.io repository
# to the Google Artifact Registry repository
gcloud-auth-docker: $(ensure-build-image)
docker run --rm $(common_mounts) $(build_tag) gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://gcr.io
docker run --rm $(common_mounts) $(build_tag) gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://us-docker.pkg.dev
Copy link
Member

@markmandel markmandel Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I forgot that I kept having to do this manually!


# Clean the gcloud configuration
clean-gcloud-config:
Expand Down
3 changes: 3 additions & 0 deletions build/includes/release.mk
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@
# targets for an Agones release
#

# agones image release registry
release_registry = us-docker.pkg.dev/agones-images/release

# generate a changelog using github-changelog-generator
gen-changelog: RELEASE_VERSION ?= $(base_version)
gen-changelog: RELEASE_BRANCH ?= main
Expand Down
2 changes: 1 addition & 1 deletion build/terraform/gke/module.tf
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ variable "pull_policy" {
}

variable "image_registry" {
default = "gcr.io/agones-images"
default = "us-docker.pkg.dev/agones-images/release"
}

variable "always_pull_sidecar" {
Expand Down
26 changes: 18 additions & 8 deletions cloudbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,28 +30,28 @@ steps:
# Restore any caches
#

- name: gcr.io/$PROJECT_ID/restore_cache
- name: us-docker.pkg.dev/$PROJECT_ID/ci/restore_cache
id: htmltest-restore-cache
args:
- '--bucket=gs://$_CACHE_BUCKET'
- '--key=$_HTMLTEST_CACHE_KEY'
waitFor: ['-']

- name: gcr.io/$PROJECT_ID/restore_cache
- name: us-docker.pkg.dev/$PROJECT_ID/ci/restore_cache
id: cpp-sdk-build-restore-cache
args:
- '--bucket=gs://$_CACHE_BUCKET'
- '--key=$_CPP_SDK_BUILD_CACHE_KEY-$( checksum sdks/cpp/cmake/prerequisites.cmake )'
waitFor: ['-']

- name: gcr.io/$PROJECT_ID/restore_cache
- name: us-docker.pkg.dev/$PROJECT_ID/ci/restore_cache
id: cpp-sdk-conformance-restore-cache
args:
- '--bucket=gs://$_CACHE_BUCKET'
- '--key=$_CPP_SDK_CONFORMANCE_CACHE_KEY-$( checksum sdks/cpp/cmake/prerequisites.cmake )'
waitFor: ['-']

- name: gcr.io/$PROJECT_ID/restore_cache
- name: us-docker.pkg.dev/$PROJECT_ID/ci/restore_cache
id: rust-sdk-build-restore-cache
args:
- '--bucket=gs://$_CACHE_BUCKET'
Expand All @@ -75,6 +75,8 @@ steps:
- name: "make-docker"
id: pull-build-image
dir: "build"
env:
- "REGISTRY=${_REGISTRY}"
args: ["pull-build-image"]
waitFor:
- build-make-docker
Expand All @@ -86,6 +88,8 @@ steps:
- name: "make-docker"
id: pull-build-sdk-base-image
dir: "build"
env:
- "REGISTRY=${_REGISTRY}"
args: ["pull-build-sdk-base-image"]
waitFor:
- build-make-docker
Expand Down Expand Up @@ -131,6 +135,8 @@ steps:
waitFor:
- lint
dir: "build"
env:
- "REGISTRY=${_REGISTRY}"
args: ["push-build-image"] # push the build image (which won't do anything if it's already there)

#
Expand All @@ -141,6 +147,8 @@ steps:
waitFor:
- ensure-build-sdk-image-base
dir: "build"
env:
- "REGISTRY=${_REGISTRY}"
args: ["push-build-sdk-base-image"]

#
Expand Down Expand Up @@ -181,7 +189,7 @@ steps:
# Cache the cpp sdk build directory, to speed up subsequent builds (considerably)
#

- name: 'gcr.io/$PROJECT_ID/save_cache'
- name: 'us-docker.pkg.dev/$PROJECT_ID/ci/save_cache'
args:
- '--bucket=gs://$_CACHE_BUCKET'
- '--key=$_CPP_SDK_BUILD_CACHE_KEY-$( checksum sdks/cpp/cmake/prerequisites.cmake )'
Expand All @@ -207,7 +215,7 @@ steps:
#
# Cache the htmltest url checks. Faster builds, and lower network flakiness.
#
- name: 'gcr.io/$PROJECT_ID/save_cache'
- name: 'us-docker.pkg.dev/$PROJECT_ID/ci/save_cache'
args:
- '--bucket=gs://$_CACHE_BUCKET'
- '--key=$_HTMLTEST_CACHE_KEY'
Expand Down Expand Up @@ -274,6 +282,8 @@ steps:
- name: "make-docker"
id: sdk-conformance
dir: "build"
env:
- "REGISTRY=${_REGISTRY}"
args: [ "-j", "5", "--output-sync=target", "run-sdk-conformance-tests"]
waitFor:
- build-images
Expand All @@ -283,7 +293,7 @@ steps:
# Cache the CPP conformance build directory, to speed up subsequent builds (considerably)
#

- name: 'gcr.io/$PROJECT_ID/save_cache'
- name: 'us-docker.pkg.dev/$PROJECT_ID/ci/save_cache'
args:
- '--bucket=gs://$_CACHE_BUCKET'
- '--key=$_CPP_SDK_CONFORMANCE_CACHE_KEY-$( checksum sdks/cpp/cmake/prerequisites.cmake )'
Expand All @@ -296,7 +306,7 @@ steps:
# Cache the Rust SDK build directory, to speed up subsequent builds (considerably)
#

- name: 'gcr.io/$PROJECT_ID/save_cache'
- name: 'us-docker.pkg.dev/$PROJECT_ID/ci/save_cache'
args:
- '--bucket=gs://$_CACHE_BUCKET'
- '--key=$_RUST_SDK_BUILD_CACHE_KEY-$( checksum test/sdk/rust/Cargo.toml )'
Expand Down
2 changes: 1 addition & 1 deletion cmd/controller/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ func parseEnvFlags() config {
}

base := filepath.Dir(exec)
viper.SetDefault(sidecarImageFlag, "gcr.io/agones-images/agones-sdk:"+pkg.Version)
viper.SetDefault(sidecarImageFlag, "us-docker.pkg.dev/agones-images/release/agones-sdk:"+pkg.Version)
viper.SetDefault(sidecarCPURequestFlag, "0")
viper.SetDefault(sidecarCPULimitFlag, "0")
viper.SetDefault(sidecarMemoryRequestFlag, "0")
Expand Down
8 changes: 4 additions & 4 deletions docs/governance/templates/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ See <a href="https://github.com/googleforgames/agones/blob/release-{version}/CHA

Images available with this release:

- [gcr.io/agones-images/agones-controller:{version}](https://gcr.io/agones-images/agones-controller:{version})
- [gcr.io/agones-images/agones-sdk:{version}](https://gcr.io/agones-images/agones-sdk:{version})
- [gcr.io/agones-images/agones-ping:{version}](https://gcr.io/agones-images/agones-ping:{version})
- [gcr.io/agones-images/agones-allocator:{version}](https://gcr.io/agones-images/agones-allocator:{version})
- [us-docker.pkg.dev/agones-images/release/agones-controller:{version}](https://us-docker.pkg.dev/agones-images/release/agones-controller:{version})
- [us-docker.pkg.dev/agones-images/release/agones-sdk:{version}](https://us-docker.pkg.dev/agones-images/release/agones-sdk:{version})
- [us-docker.pkg.dev/agones-images/release/agones-ping:{version}](https://us-docker.pkg.dev/agones-images/release/agones-ping:{version})
- [us-docker.pkg.dev/agones-images/release/agones-allocator:{version}](https://us-docker.pkg.dev/agones-images/release/agones-allocator:{version})
{ run `make release-example-image-markdown` to populate example images section below (will be more in output than in example) }
- [us-docker.pkg.dev/agones-images/examples/cpp-simple-server:{example-version}](https://us-docker.pkg.dev/agones-images/examples/cpp-simple-server:{example-version})
- [us-docker.pkg.dev/agones-images/examples/crd-client:{example-version}](https://us-docker.pkg.dev/agones-images/examples/crd-client:{example-version})
Expand Down
2 changes: 1 addition & 1 deletion install/helm/agones/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ agones:
totalRemoteAllocationTimeout: 30s
allocationBatchWaitTime: 500ms
image:
registry: gcr.io/agones-images
registry: us-docker.pkg.dev/agones-images/release
tag: 1.29.0-dev
controller:
name: agones-controller
Expand Down
2 changes: 1 addition & 1 deletion install/terraform/modules/helm3/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ variable "crd_cleanup" {
}

variable "image_registry" {
default = "gcr.io/agones-images"
default = "us-docker.pkg.dev/agones-images/release"
}

variable "pull_policy" {
Expand Down
8 changes: 4 additions & 4 deletions install/yaml/install.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14689,7 +14689,7 @@ spec:
serviceAccountName: agones-controller
containers:
- name: agones-controller
image: "gcr.io/agones-images/agones-controller:1.29.0-dev"
image: "us-docker.pkg.dev/agones-images/release/agones-controller:1.29.0-dev"
imagePullPolicy: IfNotPresent
env:
# minimum port that can be exposed to GameServer traffic
Expand All @@ -14699,7 +14699,7 @@ spec:
- name: MAX_PORT
value: "8000"
- name: SIDECAR_IMAGE # overwrite the GameServer sidecar image that is used
value: "gcr.io/agones-images/agones-sdk:1.29.0-dev"
value: "us-docker.pkg.dev/agones-images/release/agones-sdk:1.29.0-dev"
- name: ALWAYS_PULL_SIDECAR # set the sidecar imagePullPolicy to Always
value: "false"
- name: SIDECAR_CPU_REQUEST
Expand Down Expand Up @@ -14832,7 +14832,7 @@ spec:
priorityClassName: agones-system
containers:
- name: agones-ping
image: "gcr.io/agones-images/agones-ping:1.29.0-dev"
image: "us-docker.pkg.dev/agones-images/release/agones-ping:1.29.0-dev"
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
Expand Down Expand Up @@ -14905,7 +14905,7 @@ spec:
secretName: allocator-client-ca
containers:
- name: agones-allocator
image: "gcr.io/agones-images/agones-allocator:1.29.0-dev"
image: "us-docker.pkg.dev/agones-images/release/agones-allocator:1.29.0-dev"
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
Expand Down
4 changes: 4 additions & 0 deletions site/cloudbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ steps:
args: ['build', '-f', 'Dockerfile.build', '-t', 'make-docker', '.'] # we need docker and make to run everything.
- name: "make-docker"
dir: "build"
env:
- "REGISTRY=${_REGISTRY}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 what does this do? I don't think this is required, but I could be wrong?

Copy link
Collaborator Author

@gongmax gongmax Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused now. This step will do pull-build-image which use REGISTRY as part of the image tag. Currently it's still gcr.io/agones-images (Though I haven't figured out where this value come from). I think the agones-build image will be in the us-docker.pkg.dev/agones-images/ci (this change)? Though the change I made here will not work either, since I didn't specify _REGISTRY in this file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this all comes from here:

pull-build-image:

So pull and push build image targets are targets specifically for the CI system (local developers don't use them), so for those, I don't know if you want to set them to a default that will work on CI?

Also, if you have _REGISTRY you will need to specify it in the substitutions block of this cloudbuild.yaml

I'm curious if you tested running this cloud-build yaml against your dev project to see if it worked - because I would have expected it would have failed because of the missing _REGISTRY definition.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I understood they come from the build-image.mk. I'm just not sure where the REGISTRY come from in this file. So if we do not specify the REGISTRY env viable in the site/cloud-build.yaml, will it try to pull the image from gcr.io?

You're right the _REGISTRY definition is missed. I only noticed it when I saw your last comments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OIC OIC! So the REGISTRY variable in the Makefile could be set in the parent Makefile or any includes that came before it.

So in the main version of the code, it come from here:

REGISTRY ?= $(release_registry)

So if we do not specify the REGISTRY env viable in the site/cloud-build.yaml, will it try to pull the image from gcr.io?

So in your code, the REGISTRY variable is removed from the parent Makefile, so I'm not sure where it would be getting it from -- I actually expect that it would fail silently, since there wouldn't be a registry attached.

# pull a local image that may exist on a remote repository, to a local image
pull-remote-build-image:
-docker pull $(REMOTE_TAG) && docker tag $(REMOTE_TAG) $(LOCAL_TAG)

So TL;DR - I would suggest either hardcode where it's pulled from somewhere in that Makefile, or create a substitution section like we have in the main cloudbuild.yaml and pull it from there.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the details! This makes sense now.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the substitution in site/cloudbuild.yaml. Confirmed the cloud build step is looking at the us-docker.pkg.dev/${PROJECT_ID}/ci, and the corresponding agones-build image exists in us-docker.pkg.dev/agones-images/ci.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right! Okay, let's approve and hopefully nothing breaks 😀

But looks good to me!

args: ["pull-build-image"] # since we are past CI build, we can assume that the build image exists.

#
Expand Down Expand Up @@ -60,4 +62,6 @@ steps:
- GOPATH=/workspace/go
- GO111MODULE=on

substitutions:
_REGISTRY: us-docker.pkg.dev/${PROJECT_ID}/ci
tags: ["site"]
17 changes: 17 additions & 0 deletions site/content/en/docs/Guides/Client SDKs/local.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,16 @@ is to have them both run using the host network, like so:

In one shell run:

{{% feature publishVersion="1.29.0" %}}
```shell
docker run --network=host --rm us-docker.pkg.dev/agones-images/release/agones-sdk:{{< release-version >}} --local
gongmax marked this conversation as resolved.
Show resolved Hide resolved
```
{{% /feature %}}
{{% feature expiryVersion="1.29.0" %}}
```shell
docker run --network=host --rm gcr.io/agones-images/agones-sdk:{{< release-version >}} --local
```
{{% /feature %}}

You should see a similar output to what you would if you were running the binary directly, i.e. outside a container.

Expand All @@ -143,9 +150,19 @@ docker run --network=host --rm <your image here>
If you want to [mount a custom `gameserver.yaml`](#providing-your-own-gameserver-configuration-for-local-development),
this is also possible:

{{% feature publishVersion="1.29.0" %}}
```bash
wget https://raw.githubusercontent.com/googleforgames/agones/{{< release-branch >}}/examples/simple-game-server/gameserver.yaml
# required so that the `agones` user in the container can read the file
chmod o+r gameserver.yaml
docker run --network=host --rm -v $(pwd)/gameserver.yaml:/tmp/gameserver.yaml us-docker.pkg.dev/agones-images/release/agones-sdk:{{<release-version>}} --local -f /tmp/gameserver.yaml
gongmax marked this conversation as resolved.
Show resolved Hide resolved
```
{{% /feature %}}
{{% feature expiryVersion="1.29.0" %}}
```bash
wget https://raw.githubusercontent.com/googleforgames/agones/{{< release-branch >}}/examples/simple-game-server/gameserver.yaml
# required so that the `agones` user in the container can read the file
chmod o+r gameserver.yaml
docker run --network=host --rm -v $(pwd)/gameserver.yaml:/tmp/gameserver.yaml gcr.io/agones-images/agones-sdk:{{<release-version>}} --local -f /tmp/gameserver.yaml
```
{{% /feature %}}
3 changes: 2 additions & 1 deletion site/content/en/docs/Installation/Install Agones/helm.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ The following tables lists the configurable parameters of the Agones chart and t
| `agones.serviceaccount.controller` | Service account name for the controller. **Note**: Will be replaced with `agones.serviceaccount.controller.name` in Agones 1.16 | `agones-controller` |
| `agones.serviceaccount.sdk` | Service account name for the sdk. **Note**: Will be replaced with `agones.serviceaccount.sdk.name` in Agones 1.16 | `agones-sdk` |
| `agones.serviceaccount.sdk.annotations` | A map of namespaces to maps of [Annotations][annotations] added to the Agones SDK service account for the specified namespaces | `{}` |
| `agones.image.registry` | Global image registry for all images | `gcr.io/agones-images` |
| `agones.image.registry` | Global image registry for all images | `gcr.io/agones-images` |
| `agones.image.tag` | Global image tag for all images | `{{< release-version >}}` |
| `agones.image.controller.name` | Image name for the controller | `agones-controller` |
| `agones.image.controller.pullPolicy` | Image pull policy for the controller | `IfNotPresent` |
Expand Down Expand Up @@ -226,6 +226,7 @@ The following tables lists the configurable parameters of the Agones chart and t

| Parameter | Description | Default |
|---------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
| `agones.image.registry` | Global image registry for all the Agones system images | `us-docker.pkg.dev/agones-images/release`
{{% /feature %}}

[toleration]: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
Expand Down