Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WX-906 Sbt Unit Tests as Github Actions #6992

Merged
merged 137 commits into from
Feb 1, 2023
Merged
Show file tree
Hide file tree
Changes from 132 commits
Commits
Show all changes
137 commits
Select commit Hold shift + click to select a range
89fb748
workflow trial 1
Jan 18, 2023
a7f5fe7
udate
THWiseman Jan 18, 2023
c619a61
remove trigger
THWiseman Jan 18, 2023
900da1b
add workflow_dispatch
THWiseman Jan 18, 2023
41b4fc5
fix spelling mistake
THWiseman Jan 18, 2023
b742a6b
echo environment variables
THWiseman Jan 18, 2023
2c8fe94
typo
THWiseman Jan 18, 2023
ef5540f
change enviornment variable scope
THWiseman Jan 18, 2023
3960e6a
it was working the first time
THWiseman Jan 18, 2023
047b957
here goes nothing
THWiseman Jan 23, 2023
f8f73c7
getting my bearings
THWiseman Jan 23, 2023
85e949e
dotslash
THWiseman Jan 23, 2023
a028550
are you suuuuure git?
THWiseman Jan 23, 2023
9207552
checkout first
THWiseman Jan 23, 2023
49c45bf
formatting
THWiseman Jan 23, 2023
2daf2d6
try this
THWiseman Jan 23, 2023
6fc8470
steal from adam
THWiseman Jan 23, 2023
776e4a9
with scala
THWiseman Jan 23, 2023
426044a
full steal
THWiseman Jan 23, 2023
6b30ef4
v3 newer than v2
THWiseman Jan 24, 2023
61704c8
moment of truth
THWiseman Jan 24, 2023
4c70191
will this work?
THWiseman Jan 24, 2023
da89730
this is the secret sauce
THWiseman Jan 24, 2023
6c623ed
This should still be working
THWiseman Jan 25, 2023
e6cd875
one more test
THWiseman Jan 25, 2023
23b5f53
try again
THWiseman Jan 25, 2023
079b3c6
spelling
THWiseman Jan 25, 2023
b435c7a
rename file
THWiseman Jan 25, 2023
d1e7629
comments
THWiseman Jan 25, 2023
2da9d83
optimize command
THWiseman Jan 25, 2023
9d12de0
test with matrix
THWiseman Jan 25, 2023
076ed6a
try something else
THWiseman Jan 25, 2023
75c0955
tab
THWiseman Jan 25, 2023
efbc02c
more realistic workflow
THWiseman Jan 25, 2023
8bb541c
why fail fast if you can prolong your failure
THWiseman Jan 25, 2023
3476796
continue on error
THWiseman Jan 25, 2023
6b8afdb
syntax
THWiseman Jan 25, 2023
e9018d8
updated yaml
THWiseman Jan 26, 2023
f6cce2f
git secrets attempt
THWiseman Jan 26, 2023
b727570
bigger test suite
THWiseman Jan 26, 2023
a399d12
path stuff
THWiseman Jan 26, 2023
c36fd91
yolo
THWiseman Jan 26, 2023
0349d6d
name cache setup
THWiseman Jan 26, 2023
d84cbab
hyphens
THWiseman Jan 26, 2023
aee9df5
steps
THWiseman Jan 26, 2023
aaeee6f
do we even need that?
THWiseman Jan 26, 2023
adcfae8
yes we need that
THWiseman Jan 26, 2023
eda3f89
please don't break the internet
THWiseman Jan 26, 2023
81ce1da
syntax error this
THWiseman Jan 26, 2023
a985a64
desparate
THWiseman Jan 26, 2023
d4da61f
no more
THWiseman Jan 26, 2023
3132a19
case sensitive and so am I
THWiseman Jan 26, 2023
ae82349
commentos
THWiseman Jan 26, 2023
3a158ea
rename and polish
THWiseman Jan 26, 2023
4e0821c
this will not work
THWiseman Jan 30, 2023
a1b4f4b
newlines
THWiseman Jan 30, 2023
8076020
yaml
THWiseman Jan 30, 2023
d00fe60
error lol
THWiseman Jan 30, 2023
9272636
pipe
THWiseman Jan 30, 2023
2ccbc76
one step closer
THWiseman Jan 30, 2023
602c5ee
idk
THWiseman Jan 30, 2023
53df19c
syntax why
THWiseman Jan 30, 2023
cd78e00
yeye
THWiseman Jan 30, 2023
9a93df0
back to basics
THWiseman Jan 30, 2023
b691681
advanced techniques
THWiseman Jan 30, 2023
897bb7a
this wont work either
THWiseman Jan 30, 2023
025eea8
nope
THWiseman Jan 30, 2023
89a5607
try again
THWiseman Jan 30, 2023
f155b49
aisd
THWiseman Jan 30, 2023
f28381e
pipe got lost
THWiseman Jan 30, 2023
e978ad4
quotatsions
THWiseman Jan 30, 2023
e4f6f82
trying
THWiseman Jan 30, 2023
8d702de
\s
THWiseman Jan 30, 2023
778af44
rome was built from the command line
THWiseman Jan 30, 2023
7eff689
i cant keep coming up with commit messages
THWiseman Jan 30, 2023
a62a754
working?
THWiseman Jan 30, 2023
11101fb
Escape asterisks
jgainerdewar Jan 30, 2023
c4341da
Does this work?
jgainerdewar Jan 30, 2023
39e6270
Also escape dollar sign
jgainerdewar Jan 30, 2023
ceed50e
Testing
jgainerdewar Jan 30, 2023
df46ef6
More escaping
jgainerdewar Jan 30, 2023
de6704b
Less escaping
jgainerdewar Jan 30, 2023
9afc8c9
Testing
jgainerdewar Jan 30, 2023
c9640c6
EVEN MORE ESCAPING
jgainerdewar Jan 30, 2023
29cc772
What if we didn't use brackets at all
jgainerdewar Jan 30, 2023
fc01ca7
Testing
jgainerdewar Jan 30, 2023
76290cd
Testing
jgainerdewar Jan 30, 2023
c924572
Testing
jgainerdewar Jan 30, 2023
997f5a7
Testing
jgainerdewar Jan 30, 2023
3cfee4a
Testing
jgainerdewar Jan 30, 2023
0470963
Testing
jgainerdewar Jan 30, 2023
5d11931
Testing
jgainerdewar Jan 30, 2023
e83e7a4
Testing
jgainerdewar Jan 31, 2023
678f77a
Testing
jgainerdewar Jan 31, 2023
cc209c4
Testing
jgainerdewar Jan 31, 2023
069c1a6
Testing
jgainerdewar Jan 31, 2023
8368318
Testing
jgainerdewar Jan 31, 2023
9c8b38c
Remove color codes
jgainerdewar Jan 31, 2023
db128af
Try the whole thing
jgainerdewar Jan 31, 2023
461c0aa
Oops
jgainerdewar Jan 31, 2023
6358c75
Revert test changes, remove trailing comma
jgainerdewar Jan 31, 2023
52ee24f
Leave comma to deal with later, add comment
jgainerdewar Jan 31, 2023
15d9302
here goes nothing
THWiseman Jan 31, 2023
bf91a06
whitespace
THWiseman Jan 31, 2023
74796d4
using secrets
THWiseman Jan 31, 2023
247f412
comments
THWiseman Jan 31, 2023
d184b6f
yolo
THWiseman Jan 31, 2023
68fc894
indent
THWiseman Jan 31, 2023
962c04f
idgetit
THWiseman Jan 31, 2023
0302bc6
indent
THWiseman Jan 31, 2023
afe4d3b
this is dumb
THWiseman Jan 31, 2023
01fbece
copypasta
THWiseman Jan 31, 2023
37bca13
asd
THWiseman Jan 31, 2023
019f333
actionable
THWiseman Jan 31, 2023
5aa1271
whitespace
THWiseman Jan 31, 2023
e9dc5f6
remove newline:
THWiseman Jan 31, 2023
857ad95
lets seeeeee
THWiseman Jan 31, 2023
97b3666
what is even in there:
THWiseman Jan 31, 2023
7fb8069
the ol doubble commar
THWiseman Jan 31, 2023
a78c30f
the double dot
THWiseman Jan 31, 2023
f4d7216
dont ask questions
THWiseman Jan 31, 2023
54aaee3
y
THWiseman Jan 31, 2023
8403b9f
2
THWiseman Jan 31, 2023
fcf9c8f
5
THWiseman Jan 31, 2023
eaa4852
lol
THWiseman Jan 31, 2023
2609146
whitespoace
THWiseman Jan 31, 2023
f4837f8
a
THWiseman Jan 31, 2023
33fe63c
whoops
THWiseman Jan 31, 2023
df0fbc7
better formatting
THWiseman Jan 31, 2023
05ff151
commento
THWiseman Jan 31, 2023
4e338c4
eof
THWiseman Jan 31, 2023
58e8f80
length
THWiseman Jan 31, 2023
13b0062
speedtest without cache
THWiseman Jan 31, 2023
d9da2ed
cacheless speedtest
THWiseman Jan 31, 2023
e00aad3
kids call these things hashtags
THWiseman Jan 31, 2023
83b8daf
address PR feedback with liberal use of the delete key
THWiseman Jan 31, 2023
4bc0bc8
fix up some comment
THWiseman Feb 1, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .github/set_up_cromwell_action/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#This is a series of steps that will setup Java/sbt/Cromwell on the local runner.
#These steps are meant to be re-used and invoked by other Github Action Workflows
name: 'Set Up Cromwell Steps'
description: Specific steps that will set up git secrets, java, sbt, and Cromwell on the local machine.
inputs:
cromwell_repo_token: #As an input to this action, you are required to pass in a token that can be used to authenticate while checking out Cromwell.
required: true

runs:
using: "composite" # <-- this allows these steps to be used by other workflows.
steps:
#Allows this github action to use a cache to store stuff like Java and sbt files between runs.
- uses: actions/checkout@v3
name: Checkout Coursier Cache
- uses: coursier/cache-action@v6
name: Enable Coursier Cache

#Cromwell requires git-secrets be setup. Here, we set up secrets and verify success with a script.
- name: Git secrets setup
run: |
git clone https://github.com/awslabs/git-secrets.git ~/git-secrets
cd ~/git-secrets
git checkout ad82d68ee924906a0401dfd48de5057731a9bc84
sudo make install
shell: bash

- name: Secrets check
run: |
sudo ln -s "$(which echo)" /usr/local/bin/say
./minnie-kenny.sh --force
git secrets --scan-history
shell: bash

#Clone the cromwell repo to this VM.
- name: Clone Cromwell
uses: actions/checkout@v3
with:
repository: broadinstitute/cromwell
token: ${{ inputs.cromwell_repo_token }}

#Install Java to this VM. This Java version and distribution is compatible with Cromwell.
- name: Setup JDK
uses: actions/setup-java@v3
with:
distribution: temurin
java-version: 11
cache: sbt
THWiseman marked this conversation as resolved.
Show resolved Hide resolved
86 changes: 86 additions & 0 deletions .github/workflows/cromwell_unit_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
name: 'Cromwell unit tests'

#This github action runs all of Cromwell's unit tests.
#A matrix strategy is used so that each subproject can be unit tested in parallel with all other subprojects.

#This is what shows up in the github workflows page as the title.
run-name: ${{ github.actor }} running Cromwell sbt unit tests.

#What will trigger the workflow to run.
on:
workflow_dispatch: #Manual trigger from GitHub UI
push:
pull_request:
THWiseman marked this conversation as resolved.
Show resolved Hide resolved

permissions:
contents: read

jobs:
build-and-test:
strategy:
fail-fast: false #Disabling fail fast means that even if one job in the matrix fails, the others will still try to complete.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

matrix: #this array is the output of the command `sbt projects` from the cromwell root directory (minus lots of formatting)
sbtSubProject: [awsBackend,
awsS3FileSystem,
azureBlobFileSystem,
backend,
centaur,
cloud-nio-impl-drs,
cloud-nio-impl-ftp,
cloud-nio-spi,
cloud-nio-util,
cloudSupport,
common,
core,
cromiam,
cromwell-drs-localizer,
cromwellApiClient,
databaseMigration,
databaseSql,
dockerHashing,
drsFileSystem,
engine,
ftpFileSystem,
gcsFileSystem,
googlePipelinesCommon,
googlePipelinesV2Alpha1,
googlePipelinesV2Beta,
httpFileSystem,
languageFactoryCore,
perf,
server,
services,
sfsBackend,
sraFileSystem,
tesBackend,
wdlBiscayneLanguageFactory,
wdlDraft2LanguageFactory,
wdlDraft3LanguageFactory,
wdlModelBiscayne,
wdlModelDraft2,
wdlModelDraft3,
wdlNewBaseTransforms,
wdlSharedModel,
wdlSharedTransforms,
wdlTransformsBiscayne,
wdlTransformsDraft2,
wdlTransformsDraft3,
wes2cromwell,
wom,
womtool]
THWiseman marked this conversation as resolved.
Show resolved Hide resolved

#This action is using a Github free runner, rather than a Broad self-hosted one.
#This is because the Broad ones don't have sbt installed by default.
runs-on: ubuntu-latest
Comment on lines +18 to +20
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's probably fine for these small tests, that are unlikely to ever be the longest-running piece and therefore gating factor to speed.


steps:
- uses: actions/checkout@v3 # checkout the cromwell repo
- uses: ./.github/set_up_cromwell_action #Exectute this reusable github action. It will set up java/sbt/git-secrets/cromwell.
with:
cromwell_repo_token: ${{ secrets.BROADBOT_GITHUB_TOKEN }}

#Invoke SBT on a specific subproject, asking it to run all unit tests contained within that subproject.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably want to update this comment 🙂

- name: Run tests
run: |
set -e
sbt "${{ matrix.sbtSubProject }}/test"