Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fine-tune CI Workflows in PR #964

Merged
merged 10 commits into from
Aug 15, 2024
Merged

Fine-tune CI Workflows in PR #964

Changes from 5 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
890c55d
Add a CI target check job and modify the jobs to run only when the fi…
binary-ho Aug 14, 2024
df19964
Add `design` package to paths-ignore
binary-ho Aug 14, 2024
ab50d54
Add Comment for test bench test running
binary-ho Aug 14, 2024
17ca7d9
Add Comment for test `sharding test running`
binary-ho Aug 14, 2024
96bf00b
Remove All test comments
binary-ho Aug 14, 2024
9dd2ce7
Fix jobs depth
binary-ho Aug 14, 2024
a654f6f
Remove unnecessary step in benchmark job
binary-ho Aug 15, 2024
acfd1f0
Rename job name `sharding_test` to `sharding-test`
binary-ho Aug 15, 2024
cdd85ad
Update bench test targets - all bench tests dependencies
binary-ho Aug 15, 2024
57ea87b
Simplify bench target packages and remove `/api/types/**`, `/test/hel…
binary-ho Aug 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 130 additions & 88 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ on:
paths-ignore:
- 'api/docs/**'
- 'build/charts/**'
- 'design/**'
- '**/*.md'
- '**/*.txt'
- '**/.gitignore'
Expand All @@ -16,126 +17,167 @@ env:
GO_VERSION: '1.21'

jobs:
ci-target-check:
runs-on: ubuntu-latest

outputs:
build: ${{ steps.ci-target-check.outputs.build }}
bench: ${{ steps.ci-target-check.outputs.bench }}
sharding-test: ${{ steps.ci-target-check.outputs.sharding-test }}

steps:
- name: CI target check by path
uses: dorny/paths-filter@v3
id: ci-target-check
with:
filters: |
build: '**'
bench:
- 'pkg/**'
- 'server/packs/**'
sharding-test:
- 'server/backend/database/**'

build:
name: build
runs-on: ubuntu-latest

needs: ci-target-check
if: ${{ needs.ci-target-check.outputs.build == 'true' }}

steps:
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v4
with:
go-version: ${{ env.GO_VERSION }}

- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v4
with:
go-version: ${{ env.GO_VERSION }}
- name: Setup buf action
uses: bufbuild/buf-setup-action@v1
with:
github_token: ${{ github.token }}

- name: Setup buf action
uses: bufbuild/buf-setup-action@v1
with:
github_token: ${{ github.token }}
- name: Check out code
uses: actions/checkout@v4

- name: Check out code
uses: actions/checkout@v4
- name: Get tools dependencies
run: make tools

- name: Get tools dependencies
run: make tools
- name: Lint
run: make lint

- name: Lint
run: make lint
- name: Lint proto files
uses: bufbuild/buf-lint-action@v1

- name: Lint proto files
uses: bufbuild/buf-lint-action@v1
- name: Build
run: make build

- name: Build
run: make build
- name: Stack
run: docker compose -f build/docker/docker-compose.yml up --build -d

- name: Stack
run: docker compose -f build/docker/docker-compose.yml up --build -d
- name: Test
run: go test -tags integration -race -coverprofile=coverage.txt -covermode=atomic -v ./...

- name: Test
run: go test -tags integration -race -coverprofile=coverage.txt -covermode=atomic -v ./...
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
file: ./coverage.txt
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
file: ./coverage.txt
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

bench:
name: bench
runs-on: ubuntu-latest
permissions: write-all

needs: ci-target-check
if: ${{ needs.ci-target-check.outputs.bench == 'true' }}

steps:

- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v4
with:
go-version: ${{ env.GO_VERSION }}

- name: Check out code
uses: actions/checkout@v4

- name: Get tools dependencies
run: make tools

- name: Stack
run: docker compose -f build/docker/docker-compose.yml up --build -d

- name: Bench
run: make bench

- name: Download previous benchmark data
uses: actions/cache@v3
with:
path: ./cache
key: ${{ runner.os }}-benchmark

- name: Store benchmark result
uses: benchmark-action/github-action-benchmark@v1
if: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
with:
name: Go Benchmark
tool: 'go'
output-file-path: output.txt
external-data-json-path: ./cache/benchmark-data.json
fail-on-alert: false
github-token: ${{ secrets.GITHUB_TOKEN }}
comment-always: true
- name: Check Benchmark Target
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for checking changes in here also?
It seems like this job already passed this check in ci-target-check.

Copy link
Contributor Author

@binary-ho binary-ho Aug 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I sorry to not checking carefully. I will delete it.

uses: dorny/paths-filter@v3
id: bench-target-filter
with:
filters: |
bench-target:
- 'pkg/**'
- 'server/packs/**'

- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v4
with:
go-version: ${{ env.GO_VERSION }}

- name: Check out code
uses: actions/checkout@v4

- name: Get tools dependencies
run: make tools

- name: Stack
run: docker compose -f build/docker/docker-compose.yml up --build -d

- name: Bench
run: make bench

- name: Download previous benchmark data
uses: actions/cache@v3
with:
path: ./cache
key: ${{ runner.os }}-benchmark

- name: Store benchmark result
uses: benchmark-action/github-action-benchmark@v1
if: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
with:
name: Go Benchmark
tool: 'go'
output-file-path: output.txt
external-data-json-path: ./cache/benchmark-data.json
fail-on-alert: false
github-token: ${{ secrets.GITHUB_TOKEN }}
comment-always: true

sharding_test:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also change the name sharding_test to sharding-test for consistency with other naming conventions?

name: sharding_test
runs-on: ubuntu-latest

needs: ci-target-check
if: ${{ needs.ci-target-check.outputs.sharding-test == 'true' }}

steps:

- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v3
with:
go-version: ${{ env.GO_VERSION }}
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v3
with:
go-version: ${{ env.GO_VERSION }}

- name: Check out code
uses: actions/checkout@v4

- name: Check out code
uses: actions/checkout@v4
- name: Get tools dependencies
run: make tools

- name: Get tools dependencies
run: make tools
- name: Check Docker Compose Version
run: docker compose --version

- name: Check Docker Compose Version
run: docker compose --version
- name: Run the Config server, Shard 1 and Shard 2
run: docker compose -f build/docker/sharding/docker-compose.yml up --build -d --wait config1 shard1-1 shard2-1

- name: Run the Config server, Shard 1 and Shard 2
run: docker compose -f build/docker/sharding/docker-compose.yml up --build -d --wait config1 shard1-1 shard2-1
- name: Initialize the Config server
run: docker compose -f build/docker/sharding/docker-compose.yml exec config1 mongosh test /scripts/init-config1.js

- name: Initialize the Config server
run: docker compose -f build/docker/sharding/docker-compose.yml exec config1 mongosh test /scripts/init-config1.js
- name: Initialize the Shard 1
run: docker compose -f build/docker/sharding/docker-compose.yml exec shard1-1 mongosh test /scripts/init-shard1-1.js

- name: Initialize the Shard 1
run: docker compose -f build/docker/sharding/docker-compose.yml exec shard1-1 mongosh test /scripts/init-shard1-1.js

- name: Initialize the Shard 2
run: docker compose -f build/docker/sharding/docker-compose.yml exec shard2-1 mongosh test /scripts/init-shard2-1.js
- name: Initialize the Shard 2
run: docker compose -f build/docker/sharding/docker-compose.yml exec shard2-1 mongosh test /scripts/init-shard2-1.js

- name: Run the Mongos
run: docker compose -f build/docker/sharding/docker-compose.yml up --build -d --wait mongos1
- name: Run the Mongos
run: docker compose -f build/docker/sharding/docker-compose.yml up --build -d --wait mongos1

- name: Initialize the Mongos
run: docker compose -f build/docker/sharding/docker-compose.yml exec mongos1 mongosh test /scripts/init-mongos1.js
- name: Initialize the Mongos
run: docker compose -f build/docker/sharding/docker-compose.yml exec mongos1 mongosh test /scripts/init-mongos1.js

- name: Run the tests with sharding tag
run: go test -tags sharding -race -v ./test/sharding/...
- name: Run the tests with sharding tag
run: go test -tags sharding -race -v ./test/sharding/...
Loading