Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache dependencies and archive builds in all workflows #8631

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from
+894 −259
Draft

Cache dependencies and archive builds in all workflows #8631

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a50ccc7
Cache dependencies and archive builds in all workflows
dlarocque Oct 31, 2024
0fe2dd1
Use seperate node_modules for Firestore tests
dlarocque Nov 6, 2024
35f460a
Apply to all
dlarocque Nov 6, 2024
179e3b4
move chrome install earlier
dlarocque Nov 6, 2024
57fa7e6
put everything back in the firestore build
dlarocque Nov 7, 2024
45b3226
revert firestore changes but upgrade to v4 of artifact
dlarocque Nov 7, 2024
3aac731
upgrade to artifact v4 in changed-firestore
dlarocque Nov 7, 2024
f698d70
get build before node setup
dlarocque Nov 7, 2024
d8db704
move chrome installation up
dlarocque Nov 7, 2024
626b9ba
run auth tests sequentially
dlarocque Nov 7, 2024
913b735
add a bunch of test workflows
dlarocque Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions .github/workflows/canary-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,18 @@ jobs:
with:
# Canary release script requires git history and tags.
fetch-depth: 0
- name: Set up Node (20)
uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
- name: Yarn install
run: yarn
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- name: Deploy canary
run: yarn release canary
env:
Expand Down
16 changes: 11 additions & 5 deletions .github/workflows/check-changeset.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,18 @@ jobs:
with:
# This makes Actions fetch all Git history so check_changeset script can diff properly.
fetch-depth: 0
- name: Set up Node (20)
uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
- name: Yarn install
run: yarn
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- name: Run changeset script
# pull main so changeset can diff against it
run: |
Expand Down
16 changes: 11 additions & 5 deletions .github/workflows/check-docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,18 @@ jobs:
with:
# get all history for the diff
fetch-depth: 0
- name: Set up Node (20)
uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
- name: Yarn install
run: yarn
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- name: Run doc generation
run: yarn docgen:all
- name: Check for changes in docs-devsite dir (fail if so)
Expand Down
19 changes: 12 additions & 7 deletions .github/workflows/check-pkg-paths.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,19 @@ jobs:
with:
# This makes Actions fetch all Git history so run-changed script can diff properly.
fetch-depth: 0
- name: Set up Node (20)
uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
- name: Yarn install
run: yarn
- name: Yarn build
run: yarn build
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- run: yarn build
- name: Swap in public typings
run: yarn release:prepare
- name: Check paths
Expand Down
16 changes: 11 additions & 5 deletions .github/workflows/deploy-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,18 @@ jobs:
with:
# This makes Actions fetch all Git history so run-changed script can diff properly.
fetch-depth: 0
- name: Set up node (20)
uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
- name: Yarn install
run: yarn
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- name: Deploy project config if needed
run: yarn ts-node scripts/ci-test/deploy-if-needed.ts
env:
Expand Down
7 changes: 4 additions & 3 deletions .github/workflows/e2e-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,11 @@ jobs:
steps:
- name: Checkout Repo
uses: actions/checkout@v4
- name: Set up Node (20)
uses: actions/setup-node@master
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: install Chrome stable
run: |
sudo apt-get update
Expand Down
16 changes: 11 additions & 5 deletions .github/workflows/format.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,18 @@ jobs:
with:
# get all history for the diff
fetch-depth: 0
- name: Set up node (20)
uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
- name: Yarn install
run: yarn
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any metrics as to how much faster the act of caching is?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far what I've gathered is that without caching yarn install takes ~60s in CI, and the resulting node_modules is ~935MB. Restoring a cached node_modules (after a cache hit) takes only ~7s, and then the subsequent yarn install takes 5s.

I'll include more details in a section about speed changes in my PR description in a bit

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noice.

uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
DellaBitta marked this conversation as resolved.
Show resolved Hide resolved
- run: yarn install --frozen-lockfile
- name: Run formatting script
run: yarn format
- name: Check for changes (fail if so)
Expand Down
26 changes: 20 additions & 6 deletions .github/workflows/health-metrics-pull-request.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,14 +39,22 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- uses: 'google-github-actions/auth@v0'
with:
credentials_json: '${{ secrets.GCP_SA_KEY }}'
- uses: google-github-actions/setup-gcloud@v2
- run: yarn install
- run: yarn build
- name: Run health-metrics/binary-size test
run: yarn size-report
Expand All @@ -56,14 +64,20 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
node-version-file: '.nvmrc'
- uses: 'google-github-actions/auth@v0'
with:
credentials_json: '${{ secrets.GCP_SA_KEY }}'
- uses: google-github-actions/setup-gcloud@v2
- run: yarn install
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- run: yarn build
- name: Run health-metrics/modular-exports-binary-size test
run: yarn modular-export-size-report
Expand Down
16 changes: 11 additions & 5 deletions .github/workflows/lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,17 @@ jobs:

steps:
- uses: actions/checkout@v3
- name: Set up node (20)
uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
- name: yarn install
run: yarn
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- name: yarn lint
run: yarn lint
16 changes: 11 additions & 5 deletions .github/workflows/prerelease-manual-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,18 @@ jobs:
with:
# Canary release script requires git history and tags.
fetch-depth: 0
- name: Set up node (20)
uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
- name: Yarn install
run: yarn
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}
- run: yarn install --frozen-lockfile
- name: Deploy prerelease
run: yarn release custom -p ${{ github.event.inputs.prereleaseName }} -t ${{ github.event.inputs.npmTag }}
env:
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/release-log.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,9 @@ jobs:
- name: Checkout Repo
uses: actions/checkout@v4

- name: Setup Node.js 20.x
uses: actions/setup-node@master
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
node-version-file: '.nvmrc'

- name: Get PR number and send to tracker.
run: node scripts/ci/log-changesets.js
Expand Down
17 changes: 12 additions & 5 deletions .github/workflows/release-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,20 @@ jobs:
# This makes Actions fetch all Git history so that Changesets can generate changelogs with the correct commits
fetch-depth: 0

- name: Setup Node.js 20.x
uses: actions/setup-node@master
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock

- name: Install Dependencies
run: yarn
- name: Restore cached node_modules
uses: actions/cache@v4
id: node_modules
with:
path: "**/node_modules"
key: node_modules-${{ runner.arch }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}

- run: yarn install --frozen-lockfile

# Ensures a new @firebase/app is published with every release.
# This keeps the SDK_VERSION variable up to date.
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/release-prod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,19 +32,19 @@ jobs:
contents: write

steps:
- name: Set up node (20)
uses: actions/setup-node@v3
with:
node-version: 22.10.0
- name: Checkout release branch (with history)
uses: actions/checkout@v4
with:
# Release script requires git history and tags.
fetch-depth: 0
ref: ${{ github.event.inputs.release-branch }}
token: ${{ secrets.OSS_BOT_GITHUB_TOKEN }}
- name: Yarn install
run: yarn
- uses: actions/setup-node@v4
with:
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- run: yarn install --frozen-lockfile
- name: Publish to NPM
# --skipTests No need to run tests
# --skipReinstall Yarn install has already been run
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/release-staging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,6 @@ jobs:
# Block this workflow if run on a non-release branch.
if: github.event.inputs.release-branch == 'release' || endsWith(github.event.inputs.release-branch, '-releasebranch')
steps:
- name: Set up node (20)
uses: actions/setup-node@v3
with:
node-version: 22.10.0
- name: Merge main into release
uses: actions/github-script@v6
with:
Expand All @@ -68,8 +64,12 @@ jobs:
# Release script requires git history and tags.
fetch-depth: 0
ref: ${{ github.event.inputs.release-branch }}
- name: Yarn install
run: yarn
- uses: actions/setup-node@v4
with:
node-version-file: '.nvmrc'
cache: yarn
cache-dependency-path: yarn.lock
- run: yarn install --frozen-lockfile
- name: Publish to NPM
# --skipTests No need to run tests
# --skipReinstall Yarn install has already been run
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/release-tweet.yml
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot we still had this workflow, by the way, I think we can get rid of it.

Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,9 @@ jobs:
steps:
- name: Checkout Repo
uses: actions/checkout@v4
- name: Setup Node.js 20.x
uses: actions/setup-node@master
- uses: actions/setup-node@v4
with:
node-version: 22.10.0
node-version-file: '.nvmrc'
- name: Poll release notes page on devsite
run: node scripts/ci/poll_release_notes.js
env:
Expand Down
Loading
Loading