Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create workflow for monorepo CI build #18

Merged
merged 142 commits into from
Dec 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
142 commits
Select commit Hold shift + click to select a range
b69d268
Create workflow for monorepo CI build
kerbou Dec 4, 2024
013ee00
Add detect-changes
kerbou Dec 4, 2024
4304946
Fixes
kerbou Dec 4, 2024
e355350
Doh!
kerbou Dec 4, 2024
d59443b
Move detect-changes to testable action
kerbou Dec 4, 2024
dee1ecf
Fix inputs
kerbou Dec 4, 2024
508b8e0
Fix path
kerbou Dec 6, 2024
80aa7dd
kljh
kerbou Dec 6, 2024
f45b7d3
Debug print
kerbou Dec 6, 2024
c83dd1a
Doh!
kerbou Dec 6, 2024
e211d43
Will this work ?
kerbou Dec 6, 2024
9282420
lkj
kerbou Dec 6, 2024
89a794b
Introduce change
kerbou Dec 6, 2024
20663a2
ælkj
kerbou Dec 6, 2024
6fdbc41
Debug print
kerbou Dec 6, 2024
7578038
Set output vars
kerbou Dec 6, 2024
baf31fa
Forgot filter
kerbou Dec 6, 2024
a31adab
Fix output values
kerbou Dec 6, 2024
d76ae05
Remove commented-out code
kerbou Dec 6, 2024
0b01f6f
Use mountains for hostnames
kerbou Dec 6, 2024
ad8062e
Fix output variable name
kerbou Dec 6, 2024
4e64737
Reduce variance
kerbou Dec 6, 2024
94947a3
jhg
kerbou Dec 6, 2024
a02e02f
build everest
kerbou Dec 6, 2024
c3d24dd
Print debug
kerbou Dec 6, 2024
a2dfe2a
Print debug log
kerbou Dec 6, 2024
feae94e
now we're getting there
kerbou Dec 6, 2024
e106bcc
Fix plings
kerbou Dec 6, 2024
b340825
Fix title
kerbou Dec 6, 2024
fae315b
Remote write-host
kerbou Dec 6, 2024
c0f3599
Comment in montblanc and dbmigrations
kerbou Dec 6, 2024
52f0575
Run tests
kerbou Dec 6, 2024
a6afebe
Add ci_run_tests to allow_merge
kerbou Dec 6, 2024
ad764ae
New matrix
kerbou Dec 6, 2024
54a3d67
adf
kerbou Dec 6, 2024
290b953
Convert to json
kerbou Dec 6, 2024
c321692
Print JSON
kerbou Dec 6, 2024
be99a56
Compress JSON
kerbou Dec 6, 2024
aa9f6a7
lkn
kerbou Dec 6, 2024
e10a2fe
.jb
kerbou Dec 6, 2024
5794584
-klm
kerbou Dec 6, 2024
eec7fdc
æl
kerbou Dec 6, 2024
55640e9
ælj
kerbou Dec 6, 2024
06f1c20
Print debug output
kerbou Dec 6, 2024
65595aa
Build and merge artifact
kerbou Dec 6, 2024
dc8c245
Change name
kerbou Dec 6, 2024
6acdea2
Merge branch 'main' into kerbou/2607-build-monorepo-ci-for-dotnet
kerbou Dec 6, 2024
c5a26c0
Fix build error
kerbou Dec 6, 2024
4da7f66
Add comment
kerbou Dec 6, 2024
772509e
Doh !
kerbou Dec 6, 2024
4bbbac6
Remove commented-out code
kerbou Dec 6, 2024
39b3369
try this
kerbou Dec 6, 2024
1809496
Kick Nuget build
kerbou Dec 6, 2024
14a6073
Dont screw with license lines
kerbou Dec 6, 2024
210e83a
Run integration tests
kerbou Dec 9, 2024
68d11eb
Fix assemblyname
kerbou Dec 9, 2024
17bc086
Bump Nuget version to 0.14.1
kerbou Dec 9, 2024
8786de5
Run tests
kerbou Dec 9, 2024
491648a
Add project_name
kerbou Dec 9, 2024
bc40a8b
Remove debug print
kerbou Dec 9, 2024
13b61f9
Kick CI
kerbou Dec 9, 2024
0e30644
Merge branch 'main' into kerbou/2607-build-monorepo-ci-for-dotnet
kerbou Dec 9, 2024
1fa7fa3
Skip steps
kerbou Dec 9, 2024
a90438b
Kick CI
kerbou Dec 9, 2024
ba0e5d4
Enable integration tests again
kerbou Dec 9, 2024
439545b
Set relative path correct
kerbou Dec 9, 2024
88be4d1
Bump version
kerbou Dec 9, 2024
0ea1c71
Create database artifact
kerbou Dec 9, 2024
2f5d5dd
Add nuget to allow_merge
kerbou Dec 9, 2024
984cdcc
Fix cache key
kerbou Dec 9, 2024
93de3b3
Attempt to fix path
kerbou Dec 9, 2024
45f7b15
Use backslash...
kerbou Dec 9, 2024
39fbaf9
Delete dead code
kerbou Dec 9, 2024
f201c49
Kick CI
kerbou Dec 9, 2024
8f28bc3
Kick CI
kerbou Dec 9, 2024
ce74566
Fix relative path
kerbou Dec 9, 2024
3bb6724
Add dot
kerbou Dec 9, 2024
be68ab1
Add Tests
kerbou Dec 9, 2024
c62006f
Add dot
kerbou Dec 9, 2024
59d96ed
Revert paths
kerbou Dec 9, 2024
9202fa9
Pretty print matrix
kerbou Dec 10, 2024
594d77c
Build SUT project as well
kerbou Dec 10, 2024
8db0a17
Merge branch 'main' into kerbou/2607-build-monorepo-ci-for-dotnet
kerbou Dec 10, 2024
6c7647d
Fix comment
kerbou Dec 10, 2024
21c22d9
Fix subsystemtest project build failure
kerbou Dec 10, 2024
330cc12
Fix pretty print
kerbou Dec 10, 2024
22e2c34
Add debug info
kerbou Dec 10, 2024
82637fb
Elementary...
kerbou Dec 10, 2024
c4fc7c3
Print matrix
kerbou Dec 10, 2024
3ad84a6
Check that everything builds
kerbou Dec 10, 2024
3443775
Merge branch 'main' into kerbou/2607-build-monorepo-ci-for-dotnet
kerbou Dec 10, 2024
7d8e35c
Save space
kerbou Dec 10, 2024
7e2cc5b
More compression = no difference
kerbou Dec 10, 2024
f5abad2
Run tests
kerbou Dec 10, 2024
2b64e67
Bump 0.14.1 --> 0.14.2
kerbou Dec 10, 2024
742a866
Fix names
kerbou Dec 10, 2024
6af3733
Kick CI
kerbou Dec 10, 2024
a7163fd
Kick CI
kerbou Dec 10, 2024
c0f219f
Kick CI
kerbou Dec 10, 2024
ec95e07
Kick CI
kerbou Dec 10, 2024
e0586d8
Add orchestrations always
kerbou Dec 10, 2024
6c5288b
Invert if
kerbou Dec 10, 2024
4a1f979
Kick CI
kerbou Dec 10, 2024
90cdc44
Add properties
kerbou Dec 10, 2024
ce77542
Add ProcessManager to build always
kerbou Dec 10, 2024
dbc00cf
Add names
kerbou Dec 10, 2024
4c0be0c
Kick CI
kerbou Dec 10, 2024
2135bce
Move params
kerbou Dec 10, 2024
d7ee05e
Add comment
kerbou Dec 10, 2024
e1445c7
Build only changed artifacts
kerbou Dec 10, 2024
79d16b4
Merge branch 'main' into kerbou/2607-build-monorepo-ci-for-dotnet
kerbou Dec 10, 2024
08b3677
Bump 0.2.1 --> 0.2.2
kerbou Dec 10, 2024
4ca26ed
Run tests
kerbou Dec 10, 2024
31ee48b
kick ci
kerbou Dec 10, 2024
b9b3f9d
Add solutionfile
kerbou Dec 10, 2024
a9dbaf1
No need to build projects
kerbou Dec 10, 2024
eee81e9
Add pings
kerbou Dec 10, 2024
01ea4ac
Flip
kerbou Dec 10, 2024
2d2c69d
doh
kerbou Dec 10, 2024
c407d39
Skip building anything
kerbou Dec 10, 2024
c6ca96a
Kick CI
kerbou Dec 11, 2024
3ae9096
Add cachekey prefix
kerbou Dec 11, 2024
40e6199
Use same cache key
kerbou Dec 11, 2024
0397d93
Add default
kerbou Dec 11, 2024
4cb2e84
Kick CI
kerbou Dec 11, 2024
f496463
Remove comment
kerbou Dec 11, 2024
e74b95c
Remove debug print
kerbou Dec 11, 2024
53f3b54
Fix cachekeys. PR feedback
kerbou Dec 11, 2024
fbca041
Enable YAML lint
kerbou Dec 11, 2024
5b9cd0d
Fix yaml
kerbou Dec 11, 2024
bc1f40c
fix yaml
kerbou Dec 11, 2024
486d75c
Fix yaml
kerbou Dec 11, 2024
9db0666
Change names from mountains to versioned names
kerbou Dec 11, 2024
213accd
Build versions
kerbou Dec 11, 2024
6840b33
Fix jobnames
kerbou Dec 11, 2024
559c434
Fix yaml
kerbou Dec 11, 2024
b352846
Fix
kerbou Dec 11, 2024
2f075df
Adjust naming
kerbou Dec 12, 2024
7c830e4
Add names
kerbou Dec 12, 2024
3b9c3c3
Remove version tag
kerbou Dec 12, 2024
b2b7458
Merge branch 'main' into kerbou/2607-build-monorepo-ci-for-dotnet
kerbou Dec 12, 2024
ffacfac
Bump version tag
kerbou Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/actions/dotnet-affected/Find-AffectedProjects.Tests.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
Describe "Find-AffectedProjects" {
BeforeAll {
. $PSScriptRoot/Find-AffectedProjects.ps1
}

Context "When no hosts are affected" {

It 'should return nothing' {
$affectedProjectsJson = "[]"

$hostNamesJson = "{
'foo': 'Bar.csproj',
'bar': 'FooBar.csproj'
}"

$expectedResult = "{
'foo': '',
'bar': ''
}" | ConvertFrom-Json | ConvertTo-Json -Compress

$affectedHosts = Find-AffectedHosts -AffectedProjects $affectedProjectsJson -HostNames $hostNamesJson
$affectedHosts | Should -Be $expectedResult
}
}

Context "When one host is affected by changes" {

It 'should return nothing' {
$affectedProjectsJson = "[
{
'Name': 'Bar',
'Filepath': 'source/somePath/Bar.csproj'
},
{
'Name': 'BarBaz',
'Filepath': 'source/somePath/BarBaz.csproj'
}
]"

$hostNamesJson = "{
'foo': 'Bar.csproj',
'bar': 'FooBar.csproj'
}"

$expectedResult = "{
'foo': 'true',
'bar': ''
}" | ConvertFrom-Json | ConvertTo-Json -Compress

$affectedHosts = Find-AffectedHosts -AffectedProjects $affectedProjectsJson -HostNames $hostNamesJson
$affectedHosts | Should -Be $expectedResult
}
}
}
60 changes: 60 additions & 0 deletions .github/actions/dotnet-affected/Find-AffectedProjects.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<#
.SYNOPSIS
Find affected hosts given a set of affected .NET project files

.OUTPUTS
Hostnames that are affected
#>

function Find-AffectedHosts {
param (
[Parameter(Mandatory)]
$AffectedProjects,
[Parameter(Mandatory)]
$Hostnames
)

$hostnamesOutput = ($hostNames | ConvertFrom-Json)

foreach ($hostName in ($Hostnames | ConvertFrom-Json)) {
foreach ($project in $hostname.PSObject.Properties) {
$hostnamesOutput.$($project.Name) = ''

foreach ($csprojFile in $project.Value) {
foreach ($affectedProjectFilePath in ($affectedProjects | ConvertFrom-Json).Filepath) {
if ($affectedProjectFilePath.Contains($csprojFile)) {

# No less than 4 foreach loops, I know...
Write-Host "$($project.Name) is affected by a change in $affectedProjectFilePath"
$hostnamesOutput.$($project.Name) = 'true'
}
}
}
}
}

return $hostnamesOutput | ConvertTo-Json -Compress
}

<#
.SYNOPSIS

Install and run dotnet-affected
#>
function Write-AffectedProjectsFile {
param (
[Parameter(Mandatory)]
$SolutionPath,
[Parameter(Mandatory)]
$WorkspacePath,
[Parameter(Mandatory)]
$FromSha,
[Parameter(Mandatory)]
$ToSha
)

dotnet tool install dotnet-affected --global
Write-Host "dotnet affected --solution-path $solutionPath -p $workspacePath --from $fromSha --to $toSha --format traversal json"

dotnet affected --solution-path $solutionPath -p $workspacePath --from $fromSha --to $toSha --format traversal json
}
85 changes: 85 additions & 0 deletions .github/actions/dotnet-affected/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Copyright 2020 Energinet DataHub A/S
#
# Licensed under the Apache License, Version 2.0 (the "License2");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Find affected .NET projects
description: Find affected .NET projects in a solution

inputs:
solution_file:
description: The path to the solution file
required: true
workspace_path:
description: The path to the workspace
required: true
from_sha:
description: The commit SHA to compare from
required: true
to_sha:
description: The commit SHA to compare to
required: true
host_names:
description: Host names mapped to project names
required: true
print_debug_output:
description: Enable debug logging
required: false
default: "false"
outputs:
affected_changes:
description: The affected projects, if any
value: ${{ steps.filter.outputs.affected_changes }}
job_context:
description: The job context output, if any
value: ${{ steps.filter.outputs.job_context }}

runs:
using: composite
steps:
- name: Find affected projects
shell: pwsh
id: filter
run: |
. ${{ github.action_path }}/Find-AffectedProjects.ps1

$hostNamesJson = "${{ inputs.host_names }}"
$solutionPath = "${{github.workspace}}/${{inputs.solution_file}}"
$workspacePath = "${{github.workspace}}"
$fromSha = "${{ inputs.from_sha }}"
$toSha = "${{ inputs.to_sha}}"
$affectedProjectsFile = 'affected.json'

# Install and run dotnet-affected, outputs a json file with affected projects
Write-AffectedProjectsFile -SolutionPath $solutionPath -WorkspacePath $workspacePath -FromSha $fromSha -ToSha $toSha

if (Test-Path $AffectedProjectsFile) {
$affectedJson = Get-Content $AffectedProjectsFile
"affected_changes=$affectedJson" >> $env:GITHUB_OUTPUT

$job_context_output = Find-AffectedHosts -AffectedProjects $affectedJson -HostNames $hostNamesJson
"job_context=$job_context_output" >> $env:GITHUB_OUTPUT
} else {
# See https://github.com/leonardochaia/dotnet-affected?tab=readme-ov-file#dont-buildtestdeploy-when-no-projects-have-changed for details
if ($LASTEXITCODE -eq 166) {
exit 0
}
}

- name: Print debug output
shell: bash
if: ${{ inputs.print_debug_output == 'true' }}
run: |
echo "Debug output:"
echo Affected projects: '${{ steps.filter.outputs.affected_changes }}'
echo ''
echo Job context: '${{ steps.filter.outputs.job_context }}'
147 changes: 147 additions & 0 deletions .github/workflows/ci-dotnet-runtests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Copyright 2020 Energinet DataHub A/S
#
# Licensed under the Apache License, Version 2.0 (the "License2");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: CI dotnet tests

on:
workflow_call:
inputs:
affected_changes:
description: Affected projects
required: true
type: string
print_debug_output:
description: Enable debug logging
required: false
type: string
nuget_cache_key_prefix:
description: Cache key prefix
required: false
type: string
default: nuget-pr${{ github.event.pull_request.number }}
dotnet_version:
description: The version of dotnet to use
required: false
type: string
default: net8.0

jobs:
build_matrix:
name: Build matrix
runs-on: ubuntu-latest
outputs:
matrix_builds: ${{ steps.set-matrix.outputs.matrix_builds }}
matrix_tests: ${{ steps.set-matrix.outputs.matrix_tests }}
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Find testprojects in affected projects
id: set-matrix
shell: pwsh
run: |
$affectedProjects = '${{ inputs.affected_changes }}' | ConvertFrom-Json

# I couldn't find a better way to lowercase keys in a hashtable
$matrixBuilds = @()

foreach ($project in $affectedProjects) {
if ($project.Name -like '*Tests*') {
$matrixBuilds += [PSCustomObject]@{
name = $project.Name
filepath = $project.FilePath
assembly_name = "Energinet.DataHub.$($project.Name).dll"
dotnet_version = "${{ inputs.dotnet_version }}"
}
}
}

$matrixBuildsJson = $matrixBuilds | ConvertTo-Json -Compress
"matrix_builds=$matrixBuildsJson" >> $env:GITHUB_OUTPUT

print_matrix:
if: ${{ inputs.print_debug_output == 'true' }}
name: Print matrix
runs-on: ubuntu-latest
needs: build_matrix
steps:
- name: Print
shell: pwsh
run: |
Write-Host 'Print matrix'
$matrixBuilds = '${{needs.build_matrix.outputs.matrix_builds}}' | ConvertFrom-Json | ConvertTo-Json
Write-Host $matrixBuilds

# build_tests_in_project:
# name: Build ${{ matrix.project.name }}
# runs-on: ubuntu-latest
# needs: build_matrix
# strategy:
# fail-fast: false
# matrix:
# project: ${{fromJson(needs.build_matrix.outputs.matrix_builds)}}
# env:
# OUTPUT_PATH: ${{ github.workspace }}/source/${{ matrix.project.name}}/bin/Release/${{matrix.project.dotnet_version}}
# steps:
# - name: Checkout repository
# uses: actions/checkout@v4

# - uses: actions/cache@v4
# with:
# path: ~/.nuget/packages
# key: ${{ inputs.nuget_cache_key_prefix }}-${{ runner.os }}-${{ hashFiles('**/*.csproj') }}

# - name: Build ${{ matrix.project.name }}
# run: |
# dotnet publish \
# '${{ matrix.project.filepath }}' \
# --configuration Release \
# --output '${{ env.OUTPUT_PATH }}' \
# -p:ErrorOnDuplicatePublishOutputFiles=false # To avoid the error "Found multiple publish output files with the same relative path"

# # ls ${{ env.OUTPUT_PATH}}

# - uses: actions/upload-artifact@v4
# with:
# retention-days: 2
# name: pm-tests-${{ matrix.project.name }}-${{ github.event.pull_request.number }}
# path: ${{ env.OUTPUT_PATH }}/*
kerbou marked this conversation as resolved.
Show resolved Hide resolved

integration_tests:
needs: [
build_matrix
]
strategy:
fail-fast: false
matrix:
project: ${{fromJson(needs.build_matrix.outputs.matrix_builds)}}
uses: Energinet-DataHub/.github/.github/workflows/dotnet-postbuild-test-monorepo.yml@v14
with:
azure_integrationtest_tenant_id: ${{ vars.integration_test_azure_tenant_id }}
azure_integrationtest_subscription_id: ${{ vars.integration_test_azure_subscription_id }}
azure_integrationtest_spn_id: ${{ vars.integration_test_azure_spn_id_oidc }}
azure_keyvault_url: ${{ vars.integration_test_azure_keyvault_url }}
environment: AzureAuth
run_integration_tests: true
tests_filter_expression: empty # Means skip.
use_azure_functions_tools: true
aspnetcore_test_contentroot_variable_name: empty
aspnetcore_test_contentroot_variable_value: empty
solution_file_path: source\ProcessManager.sln
nuget_cache_key_prefix: ${{ inputs.nuget_cache_key_prefix }}

# Matrix parameters
testproject_artifact_name: pm-tests-${{ matrix.project.name }}-${{ github.event.pull_request.number }}
testproject_name: ${{ matrix.project.name }}
tests_dll_file_path: \source\${{ matrix.project.name}}\bin\Release\${{matrix.project.dotnet_version}}\${{ matrix.project.assembly_name }}
Loading
Loading