Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker: Update to work with corepack and yarn v4 #9764

Merged
merged 18 commits into from
Dec 28, 2023
Merged

Docker: Update to work with corepack and yarn v4 #9764

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
9dfe31b
Docker: Remove COPY steps no longer needed with yarn v4
Tobbe Dec 27, 2023
e54071b
More COPYs
Tobbe Dec 27, 2023
910150b
Enable corepack and create needed dir
Tobbe Dec 27, 2023
0b5e19c
Make sure packageManager is set
Tobbe Dec 27, 2023
40a7bd9
Merge branch 'main' into tobbe-docker-yarn-v4
Tobbe Dec 27, 2023
d602801
Fix naming and comment api-web
Tobbe Dec 27, 2023
9ddcad3
Get started on updating docker docs
Tobbe Dec 27, 2023
3528cc4
More docs updates
Tobbe Dec 27, 2023
e3d297f
doc edits
jtoar Dec 28, 2023
69bc5cc
doc edits
jtoar Dec 28, 2023
8a85658
doc edits
jtoar Dec 28, 2023
34cea4c
update info to index, update jq steps
jtoar Dec 28, 2023
6a66d17
refactor out jq step
jtoar Dec 28, 2023
ac03f45
add python comment
jtoar Dec 28, 2023
fd6d78a
update docs
jtoar Dec 28, 2023
5e41153
update docs
jtoar Dec 28, 2023
a92eaf6
Merge branch 'main' into tobbe-docker-yarn-v4
jtoar Dec 28, 2023
e323110
link to the forums
jtoar Dec 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 41 additions & 28 deletions docs/docs/docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ yarn rw experimental setup-docker

The setup commands does several things:
- writes four files: `Dockerfile`, `.dockerignore`, `docker-compose.dev.yml`, and `docker-compose.prod.yml`
- adds the official yarn [workspace-tools](https://v3.yarnpkg.com/features/plugins#official-plugins) plugin to configure yarn with the ability to only install production dependencies
- adds the `@redwoodjs/api-server` and `@redwoodjs/web-server` packages to the api and web sides respectively
- edits the `browser.open` setting in the `redwood.toml` (right now, if it's set to `true`, it'll break the dev server when running the `docker-compose.dev.yml`)

Expand All @@ -45,6 +44,8 @@ docker compose -f ./docker-compose.prod.yml up

If your api side or web side depend on env vars at build time, you may need to supply them as `--build-args`, or in the compose files.

This is often the most tedious part of setting up Docker. Have ideas of how it could be better? Let us know on the [forums](https://community.redwoodjs.com/)!

:::

The first time you do this, you'll have to use the `console` stage to go in and migrate the database—just like you would with a Redwood app on your machine:
Expand Down Expand Up @@ -72,7 +73,6 @@ FROM node:20-bookworm-slim as base

We use a Node.js 20 image as the base image because that's the version Redwood targets.
"bookworm" is the codename for the current stable distribution of Debian (version 12).
We think it's important to pin the version of the OS just like we pin the version of Node.js.
Lastly, the "slim" variant of the `node:20-bookworm` image only includes what Node.js needs which reduces the image's size while making it more secure.

:::tip Why not alpine?
Expand All @@ -81,10 +81,19 @@ While alpine may be smaller, it uses musl, a different C standard library.
In developing this Dockerfile, we prioritized security over size.

If you know what you're doing feel free to change this—it's your Dockerfile now!
Just remember to change the `apt-get` instructions below too if needed.
Just remember to change the `apt-get` instructions further down too if needed.

:::

Moving on, next we have `corepack enable`:

```Dockerfile
RUN corepack enable
```

[Corepack](https://nodejs.org/docs/latest-v18.x/api/corepack.html), Node's manager for package managers, needs to be enabled so that Yarn can use the `packageManager` field in your project's root `package.json` to pick the right version of itself.
If you'd rather check in the binary, you still can, but you'll need to remember to copy it over (i.e. `COPY --chown=node:node .yarn/releases .yarn/releases`).

```Dockerfile
RUN apt-get update && apt-get install -y \
openssl \
Expand All @@ -94,8 +103,8 @@ RUN apt-get update && apt-get install -y \

The `node:20-bookworm-slim` image doesn't have [OpenSSL](https://www.openssl.org/), which [seems to be a bug](https://github.com/nodejs/docker-node/issues/1919).
(It was included in the "bullseye" image, the codename for Debian 11.)
On Linux, [Prisma needs OpenSSL](https://www.prisma.io/docs/reference/system-requirements#linux-runtime-dependencies).
We install it, and Python and its dependencies are there ready to be uncommented if you need them. See the [Troubleshooting](#python) section for more.
On Linux, [Prisma needs OpenSSL](https://www.prisma.io/docs/reference/system-requirements#linux-runtime-dependencies), so we install it here via Ubuntu's package manager APT.
Python and its dependencies are there ready to be uncommented if you need them. See the [Troubleshooting](#python) section for more information.

[It's recommended](https://docs.docker.com/develop/develop-images/instructions/#apt-get) to combine `apt-get update` and `apt-get install -y` in the same `RUN` statement for cache busting.
After installing, we clean up the apt cache to keep the layer lean. (Running `apt-get clean` isn't required—[official Debian images do it automatically](https://github.com/moby/moby/blob/03e2923e42446dbb830c654d0eec323a0b4ef02a/contrib/mkimage/debootstrap#L82-L105).)
Expand All @@ -106,14 +115,12 @@ USER node

This and subsequent `chown` options in `COPY` instructions are for security.
[Services that can run without privileges should](https://docs.docker.com/develop/develop-images/instructions/#user).
The Node.js image includes a user, `node`, created with an explicit `uid` and `gid`.
The Node.js image includes a user, `node`, created with an explicit `uid` and `gid` (`1000`).
We reuse it.

```Dockerfile
WORKDIR /home/node/app

COPY --chown=node:node .yarn/plugins .yarn/plugins
COPY --chown=node:node .yarn/releases .yarn/releases
COPY --chown=node:node .yarnrc.yml .
COPY --chown=node:node package.json .
COPY --chown=node:node api/package.json api/
Expand All @@ -123,11 +130,13 @@ COPY --chown=node:node yarn.lock .

Here we copy the minimum set of files that the `yarn install` step needs.
The order isn't completely arbitrary—it tries to maximize [Docker's layer caching](https://docs.docker.com/build/cache/).
We expect `yarn.lock` to change more than the package.json files, the package.json files to change more than `.yarnrc.yml` , and `.yarnrc.yml` to change more than the binary, etc.
We expect `yarn.lock` to change more than the `package.json`s and the `package.json`s to change more than `.yarnrc.yml`.
That said, it's hard to argue that these files couldn't be arranged differently, or that the `COPY` instructions couldn't be combined.
The important thing is that they're all here, before the `yarn install` step:

```Dockerfile
RUN mkdir -p /home/node/.yarn/berry/index

RUN --mount=type=cache,target=/home/node/.yarn/berry/cache,uid=1000 \
--mount=type=cache,target=/home/node/.cache,uid=1000 \
CI=1 yarn install
Expand All @@ -137,6 +146,8 @@ This step installs all your project's dependencies—production and dev.
Since we use multi-stage builds, your production images won't pay for the dev dependencies installed in this step.
The build stages need the dev dependencies.

The `mkdir` step is a workaround for a permission error. We're working on removing it, but for now if you remove it the install step will probably fail.

This step is a bit more involved than the others.
It uses a [cache mount](https://docs.docker.com/build/cache/#use-your-package-manager-wisely).
Yarn operates in three steps: resolution, fetch, and link.
Expand All @@ -145,11 +156,8 @@ We could disable it all together, but by using a cache mount, we can still get t
We set it to the default directory here, but you can change its location in `.yarnrc.yml`.
If you've done so you'll have to change it here too.

The last thing to note is that we designate the node user.
[The node user's `uid` is `1000`](https://github.com/nodejs/docker-node/blob/57d57436d1cb175e5f7c8d501df5893556c886c2/18/bookworm-slim/Dockerfile#L3-L4).

One more thing to note: without setting `CI=1`, depending on the deploy provider, yarn may think it's in a TTY, making the logs difficult to read. With this set, yarn adapts accordingly.
Enabling CI enables [immutable installs](https://v3.yarnpkg.com/configuration/yarnrc#enableImmutableInstalls) and [inline builds](https://v3.yarnpkg.com/configuration/yarnrc#enableInlineBuilds), both of which are highly recommended. For more information on those settings:
Enabling CI enables [immutable installs](https://v3.yarnpkg.com/configuration/yarnrc#enableImmutableInstalls) and [inline builds](https://v3.yarnpkg.com/configuration/yarnrc#enableInlineBuilds), both of which are highly recommended.

```Dockerfile
COPY --chown=node:node redwood.toml .
Expand All @@ -161,12 +169,14 @@ We'll need these config files for the build and production stages.
The `redwood.toml` file is Redwood's de-facto config file.
Both the build and serve stages read it to enable and configure functionality.

`.env.defaults` is ok to include.
This file is committed to git, but `.env` is not.
:::warning `.env.defaults` is ok to include but `.env` is not

If you add a secret to the Dockerfile, it can be excavated.
While it's technically true that multi stage builds add a sort of security layer, it's not a best practice.
Leave them out and look to your deploy provider for further configuration.

:::

### The `api_build` stage

The `api_build` stage builds the api side:
Expand All @@ -192,6 +202,8 @@ The `api_serve` stage serves your GraphQL api and functions:
```Dockerfile
FROM node:20-bookworm-slim as api_serve

RUN corepack enable

RUN apt-get update && apt-get install -y \
openssl \
# python3 make gcc \
Expand All @@ -206,27 +218,25 @@ Docker's [multi-stage builds](https://docs.docker.com/build/building/multi-stage
USER node
WORKDIR /home/node/app

COPY --chown=node:node .yarn/plugins .yarn/plugins
COPY --chown=node:node .yarn/releases .yarn/releases
COPY --chown=node:node .yarnrc.yml .yarnrc.yml
COPY --chown=node:node api/package.json .
COPY --chown=node:node package.json .
COPY --chown=node:node api/package.json api/
COPY --chown=node:node yarn.lock yarn.lock
```

The thing that's easy to miss here is that we're copying the `api/package.json` file into the base directory, so that it's just `package.json` in the image.
This is for the production `yarn install` in the next step.

Like other `COPY` instructions, ordering these files with care enables layering caching.

```Dockerfile
RUN mkdir -p /home/node/.yarn/berry/index

RUN --mount=type=cache,target=/home/node/.yarn/berry/cache,uid=1000 \
--mount=type=cache,target=/home/node/.cache,uid=1000 \
CI=1 yarn workspaces focus api --production
```

This is a critical step for image size.
We don't use the regular `yarn install` command.
Using the [official workspaces plugin](https://github.com/yarnpkg/berry/tree/master/packages/plugin-workspace-tools)—which will be included by default in yarn v4—we "focus" on the api workspace, only installing its production dependencies.
Using the [official workspaces plugin](https://github.com/yarnpkg/berry/tree/master/packages/plugin-workspace-tools)—which is included by default in yarn v4—we "focus" on the api workspace, only installing its production dependencies.

The cache mount will be populated at this point from the install in the `base` stage, so the fetch step should fly by.

Expand All @@ -241,9 +251,9 @@ COPY --chown=node:node --from=api_build /home/node/app/node_modules/.prisma /hom
```

Here's where we really take advantage of multi-stage builds by copying from the `api_build` stage.
All the building has been done for us—now we can just grab the artifacts without having to lug around the dev dependencies.
At this point all the building has been done. Now we can just grab the artifacts without having to lug around the dev dependencies.

There's one more thing that was built—the prisma client in `node_modules/.prisma`.
There's one more thing that was built: the prisma client in `node_modules/.prisma`.
We need to grab it too.

```Dockerfile
Expand All @@ -255,7 +265,7 @@ CMD [ "node_modules/.bin/rw-server", "api", "--load-env-files" ]
Lastly, the default command is to start the api server using the bin from the `@redwoodjs/api-server` package.
You can override this command if you have more specific needs.

Note that the Redwood CLI isn't available anymore.
Note that the Redwood CLI isn't available anymore. (It's a dev dependency.)
To access the server bin, we have to find its path in `node_modules`.
Though this is somewhat discouraged in modern yarn, since we're using the `node-modules` node linker, it's in `node_modules/.bin`.

Expand All @@ -267,7 +277,7 @@ This `web_build` builds the web side:
FROM base as web_build

COPY --chown=node:node web web
RUN node_modules/.bin/redwood build web --no-prerender
RUN yarn redwood build web --no-prerender
```

After the work we did in the base stage, building the web side amounts to copying in the web directory and running `yarn redwood build web`.
Expand Down Expand Up @@ -299,15 +309,18 @@ The key line here is the first one—this stage uses the `api_build` stage as it
```Dockerfile
FROM node:20-bookworm-slim as web_serve

RUN corepack enable

USER node
WORKDIR /home/node/app

COPY --chown=node:node .yarn/plugins .yarn/plugins
COPY --chown=node:node .yarn/releases .yarn/releases
COPY --chown=node:node .yarnrc.yml .
COPY --chown=node:node package.json .
COPY --chown=node:node web/package.json .
COPY --chown=node:node yarn.lock .

RUN mkdir -p /home/node/.yarn/berry/index

RUN --mount=type=cache,target=/home/node/.yarn/berry/cache,uid=1000 \
--mount=type=cache,target=/home/node/.cache,uid=1000 \
CI=1 yarn workspaces focus web --production
Expand Down
27 changes: 19 additions & 8 deletions packages/cli/src/commands/experimental/templates/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@
# ------------------------------------------------
FROM node:20-bookworm-slim as base

RUN corepack enable

# We tried to make the Dockerfile as lean as possible. In some cases, that means we excluded a dependency your project needs.
# By far the most common is Python. If you're running into build errors because `python3` isn't available,
# uncomment the line below here and in other stages as necessary:
RUN apt-get update && apt-get install -y \
openssl \
# python3 make gcc \
Expand All @@ -10,14 +15,14 @@ RUN apt-get update && apt-get install -y \
USER node
WORKDIR /home/node/app

COPY --chown=node:node .yarn/plugins .yarn/plugins
COPY --chown=node:node .yarn/releases .yarn/releases
COPY --chown=node:node .yarnrc.yml .
COPY --chown=node:node package.json .
COPY --chown=node:node api/package.json api/
COPY --chown=node:node web/package.json web/
COPY --chown=node:node yarn.lock .

RUN mkdir -p /home/node/.yarn/berry/index

RUN --mount=type=cache,target=/home/node/.yarn/berry/cache,uid=1000 \
--mount=type=cache,target=/home/node/.cache,uid=1000 \
CI=1 yarn install
Expand Down Expand Up @@ -56,6 +61,8 @@ RUN yarn redwood build web --no-prerender
# ------------------------------------------------
FROM node:20-bookworm-slim as api_serve

RUN corepack enable

RUN apt-get update && apt-get install -y \
openssl \
# python3 make gcc \
Expand All @@ -64,12 +71,13 @@ RUN apt-get update && apt-get install -y \
USER node
WORKDIR /home/node/app

COPY --chown=node:node .yarn/plugins .yarn/plugins
COPY --chown=node:node .yarn/releases .yarn/releases
COPY --chown=node:node .yarnrc.yml .
COPY --chown=node:node api/package.json .
COPY --chown=node:node package.json .
COPY --chown=node:node api/package.json api/
COPY --chown=node:node yarn.lock .

RUN mkdir -p /home/node/.yarn/berry/index

RUN --mount=type=cache,target=/home/node/.yarn/berry/cache,uid=1000 \
--mount=type=cache,target=/home/node/.cache,uid=1000 \
CI=1 yarn workspaces focus api --production
Expand All @@ -90,15 +98,18 @@ CMD [ "node_modules/.bin/rw-server", "api", "--load-env-files" ]
# ------------------------------------------------
FROM node:20-bookworm-slim as web_serve

RUN corepack enable

USER node
WORKDIR /home/node/app

COPY --chown=node:node .yarn/plugins .yarn/plugins
COPY --chown=node:node .yarn/releases .yarn/releases
COPY --chown=node:node .yarnrc.yml .
COPY --chown=node:node web/package.json .
COPY --chown=node:node package.json .
COPY --chown=node:node web/package.json web/
COPY --chown=node:node yarn.lock .

RUN mkdir -p /home/node/.yarn/berry/index

RUN --mount=type=cache,target=/home/node/.yarn/berry/cache,uid=1000 \
--mount=type=cache,target=/home/node/.cache,uid=1000 \
CI=1 yarn workspaces focus web --production
Expand Down
Loading