Concurrency Tag (#398)

* Version 15.5.0
- *NEW:* For muti-database target builds, you can now specify custom concurrency tag. Previously, the only concurrency differentitor was by SQL Server Name. Please see the docs on [Concurrency](/docs/concurrency_options.md) and [Database targeting options](docs/override_options.md) to understand how to use this new feature.
- *UPDATED:* DACPAC creation timeouts now set to the value of `--defaultscripttimeout`. Previously, it was using the default settings.
- *UPDATED:* If a script package is failing after retries due to a timeout, the build will now immediately fail. The prior behavior was to create a custom DACPAC (if configured) and continue trying the build. This was just delaying the inevitable failure and wasting time.

.github/workflows/container-build.yml

@@ -34,7 +34,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       -
         name: Build and push
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v5
         with:
           context: "{{defaultContext}}:src"
           platforms: linux/amd64,linux/arm64

CHANGELOG.md

@@ -1,5 +1,10 @@
 # SQL Build Manager Change Log
 
+### Version 15.5.0
+- *NEW:* For muti-database target builds, you can now specify custom concurrency tag. Previously, the only concurrency differentitor was by SQL Server Name. Please see the docs on [Concurrency](/docs/concurrency_options.md) and [Database targeting options](docs/override_options.md) to understand how to use this new feature.
+- *UPDATED:* DACPAC creation timeouts now set to the value of `--defaultscripttimeout`. Previously, it was using the default settings.
+- *UPDATED:* If a script package is failing after retries due to a timeout, the build will now immediately fail. The prior behavior was to create a custom DACPAC (if configured) and continue trying the build. This was just delaying the inevitable failure and wasting time.
+
 ### Version 15.4.2
 - *UPDATED:* Converted Batch Node Pool creation to the new `Azure.ResourceManager.Batch` SDK. **NOTE:** This may require you to add batch specific NSG rules if you deploy into a subnet. See [network.bicep's](./scripts/templates/Modules/network.bicep) `nsgBatchResource` resource to see the rules that are needed.
 

docs/concurrency_options.md

@@ -1,4 +1,4 @@
-# Concurrency Options for Threaded, Batch and Kubernetes executions
+# Concurrency Options for Threaded, Batch, Kubernetes, Container App and ACI executions
 
 You can control the level of parallel execution with the combination of two arguments: `--concurrency` and `--concurrencytype`. While their meaning for threaded and batch/kubernetes are similar, there are some distinctions and subtleties when used together
 
@@ -19,12 +19,19 @@ You can control the level of parallel execution with the combination of two argu
 - `Count` - (default) will use the value for `concurrency` as the maximum number of concurrent tasks allowed
 - `Server` - When using this value, the `concurrency` value is ignored. Instead, the app will interrogate the database targets and allow one task per SQL Server at a time
 
-    _For example:_ if there are 5 unuque SQL Server targets in the database targets config, there will 1 task per server, up to 5 concurrent tasks total
+    _For example:_ if there are 5 unique SQL Server targets in the database targets config, there will 1 task per server, up to 5 concurrent tasks total
 
 - `MaxPerServer` - Will interrogate the database targets and allow multiple tasks per server, up to the `concurrency` value.
 
     _For example:_ If there are 5 unique SQL Server targets in the database targets config file and the `concurrency` flag is set to 3, then it will run 3 tasks per server at a time, up to 15 concurrent tasks (5 servers * 3 tasks per server).
 
+- `Tag` - Same behavior as `Server` but allows you to set a concurrency value per tag value instead of server. See [database targets config file](override_options.md) for details on setting the concurrency tag value.
+
+- `MaxPerTag` -  - Same behavior as `MaxPerServer` but allows you to set a concurrency value per tag value instead of server.  See [database targets config file](override_options.md) for details on setting the concurrency tag value
+
+*NOTE:* If a concurrency typf of `Tag` or `MaxPerTag` is set but there are database targets that are missing a tag value, the build will not start. 
+
+
 ### --concurrency
 
 This argument takes an integer number (default is 8) defining the maximum number of concurrent threads per the set `concurrencytype`

docs/override_options.md

@@ -31,6 +31,21 @@ sqllab1.database.windows.net:client,SqlBuildTest009
 sqllab1.database.windows.net:client,SqlBuildTest010
 ```
 
+Optionally, if you need to set a concurrency value other that the target server (see [Concurrency Options](concurrency_options.md) for how to leverage the `Tag` and `MaxPerTag` settings), you can add a tag value to the line in the format of:
+
+``` bash
+<server name>:client,<target database>#<tag value>
+```
+
+for example:
+
+``` bash
+sqllab1.database.windows.net:client,SqlBuildTest007#Tag1
+sqllab1.database.windows.net:client,SqlBuildTest008#Tag1
+sqllab1.database.windows.net:client,SqlBuildTest009#Tag2
+sqllab1.database.windows.net:client,SqlBuildTest010#Tag2
+```
+
 Why `client`?  Inside the `.sbm` file, there is a default database target set to `client`. If you don't provide an override, it will look for a database of that name. So, that combination of `client,target` tells the builder to substitute the database name `client` with the database name `target` at runtime.
 
  This is why the flag name is `--override`!
@@ -42,7 +57,11 @@ Why `client`?  Inside the `.sbm` file, there is a default database target set to
 
  ``` sql
  SELECT [ServerName] + ':client,' + [DatabaseName]
- ```
+
+ or
+
+  SELECT [ServerName] + ':client,' + [DatabaseName] + '#' + [TagValue]
+  ```
 
 
 ### Runtime

scripts/templates/Database/create_database_override_files.ps1

@@ -1,8 +1,7 @@
 param
 (
     [string] $path,
-    [string] $prefix,
-    [string] $resourceGroupName
+    [string] $prefix
 )
 
 . ./../prefix_resource_names.ps1 -prefix $prefix
@@ -16,7 +15,11 @@ $outputDbConfigFile = Join-Path $path "databasetargets.cfg"
 $databaseDbWithBadTargetConfigFile = Join-Path $path "databasetargets-badtargets.cfg"
 $clientDbConfigFile = Join-Path $path "clientdbtargets.cfg"
 $doubleClientDbConfigFile = Join-Path $path "clientdbtargets-doubledb.cfg"
+$taggedDbConfigFile = Join-Path $path "databasetargets-tag.cfg"
+$taggedClientDbConfigFile = Join-Path $path "clientdbtargets-tag.cfg"
 $serverTextFile = Join-Path $path "server.txt"
+$tag = @("TagA","TagB","TagC")
+$counter = 0
 
 $sqlServers =  (az sql server list --resource-group $resourceGroupName ) | ConvertFrom-Json
 Write-Host "Using server targets: $sqlServers"  -ForegroundColor Cyan
@@ -28,10 +31,19 @@ foreach($server in $sqlServers)
     {
         if($db -ne "master")
         {
+            if($counter -gt 2)
+            {
+                $counter = 0
+            }
             $outputDbConfig += ,@($server.fullyQualifiedDomainName + ":SqlBuildTest,"+$db) 
             $databaseDbWithBadTargetConfig += ,@($server.fullyQualifiedDomainName + ":SqlBuildTest,"+$db) 
+            $taggedDbConfig += ,@($server.fullyQualifiedDomainName + ":SqlBuildTest,"+$db +"#" + $tag[$counter])
+            $taggedClientDbConfig += ,@($server.fullyQualifiedDomainName + ":client,"+$db +"#" + $tag[$counter])
             $clientDbConfig += ,@($server.fullyQualifiedDomainName + ":client,"+$db) 
+            $counter = $counter +1; 
         }
+
+
     }
     $databaseDbWithBadTargetConfig += ,@($server.fullyQualifiedDomainName + ":SqlBuildTest,ThisIsABadDbName") 
 }
@@ -52,18 +64,24 @@ foreach($server in $sqlServers)
 }
     <# Action that will repeat until the condition is met #>
 
-Write-Host "Writing test database config to  path set to $outputDbConfigFile" -ForegroundColor DarkGreen
+Write-Host "Writing test database config to   $outputDbConfigFile" -ForegroundColor DarkGreen
 $outputDbConfig | Set-Content -Path $outputDbConfigFile
 
-Write-Host "Writing test database config to  path set to $clientDbConfigFile" -ForegroundColor DarkGreen
+Write-Host "Writing test database config to path $clientDbConfigFile" -ForegroundColor DarkGreen
 $clientDbConfig | Set-Content -Path $clientDbConfigFile
 
-Write-Host "Writing test database config to  path set to $doubleClientDbConfigFile" -ForegroundColor DarkGreen
+Write-Host "Writing test database config with tags to path $taggedClientDbConfigFile" -ForegroundColor DarkGreen
+$taggedClientDbConfigFile | Set-Content -Path $taggedClientDbConfigFile
+
+Write-Host "Writing test database config to path $doubleClientDbConfigFile" -ForegroundColor DarkGreen
 $doubleClientDbConfig | Set-Content -Path $doubleClientDbConfigFile
 
-Write-Host "Writing test database config to  path set to $databaseDbWithBadTargetConfigFile" -ForegroundColor DarkGreen
+Write-Host "Writing test database config to path $databaseDbWithBadTargetConfigFile" -ForegroundColor DarkGreen
 $databaseDbWithBadTargetConfig | Set-Content -Path $databaseDbWithBadTargetConfigFile
 
+Write-Host "Writing test database config with tags to path $taggedDbConfigFile" -ForegroundColor DarkGreen
+$taggedDbConfig | Set-Content -Path $taggedDbConfigFile
 
 Write-Host "Creating server.txt file for SQL Query override config tests" -ForegroundColor DarkGreen
-$sqlServers[0].fullyQualifiedDomainName.trim()   | Set-Content -Path $serverTextFile
+$sqlServers[0].fullyQualifiedDomainName.trim()   | Set-Content -Path $serverTextFile
+

scripts/templates/Modules/aks.bicep

@@ -35,11 +35,11 @@ resource aksAcrPull 'Microsoft.Authorization/roleAssignments@2020-04-01-preview'
   }
 }
 
-resource aks 'Microsoft.ContainerService/managedClusters@2022-05-02-preview' = {
+resource aks 'Microsoft.ContainerService/managedClusters@2023-05-01' = {
   name: aksClusterName
   location: location
   sku:{
-    name: 'Basic'
+    name: 'Base'
     tier: 'Free'
   }
   identity: {

scripts/templates/create_all_settingsfiles_fromprefix.ps1

@@ -22,4 +22,5 @@ param (
 
 ./aci/create_aci_settingsfile_fromprefix.ps1 -sbmExe $sbmExe -path $outputPath -resourceGroupName $resourceGroupName -prefix $prefix  
 
-./Database/create_database_override_files.ps1 -sbmExe $sbmExe -path $outputPath -resourceGroupName $resourceGroupName
+. ./prefix_resource_names.ps1 -prefix $prefix
+./Database/create_database_override_files.ps1 -path $outputPath -prefix $prefix 

scripts/templates/kubernetes/create_aks_settingsfile_fromprefix.ps1

@@ -50,3 +50,8 @@ Write-Host "Using keyvault name: '$keyVaultName'" -ForegroundColor DarkGreen
 $scriptDir = Split-Path $script:MyInvocation.MyCommand.Path
 .$scriptDir/create_aks_settingsfile.ps1 -sbmExe $sbmExe -path $path -resourceGroupName  $resourceGroupName -storageAccountName $storageAccountName -eventHubNamespaceName $eventHubNamespaceName -serviceBusNamespaceName $serviceBusNamespaceName -sqlUserName $sqlUserName -sqlPassword $sqlPassword -acrName $containerRegistryName  -keyVaultName $keyVaultName -serviceAccount $serviceAccountName -podCount $podCount
 
+$yaml = Join-Path -Path $scriptDir -ChildPath "runnerpod.yaml"
+$destYaml = Join-Path -Path $path -ChildPath "runnerpod.yaml"
+(Get-Content $yaml) | foreach-object { $_ -replace "<<prefix>>", $prefix } | Set-Content $destYaml
+
+

scripts/templates/kubernetes/runnerpod.yaml

@@ -0,0 +1,28 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: sbm-runner
+  namespace: sqlbuildmanager
+  labels:
+    azure.workload.identity/use: 'true'
+spec:
+  nodeSelector:
+    kubernetes.io/os: linux
+  serviceAccountName: <<prefix>>serviceaccount
+  containers:
+  - name:  sbm-runner
+    image: <<prefix>>containerregistry.azurecr.io/sqlbuildmanager:latest-vNext
+    imagePullPolicy: Always
+    resources:
+      limits:
+        memory: 512M
+        cpu: 500m
+    command:
+    - sh
+    - -c
+    - tail -f /dev/null 
+  restartPolicy: Always
+
+
+#kubectl get pods -n sqlbuildmanager
+#kubectl exec --stdin --tty  sbm-runner -n sqlbuildmanager -- /bin/bash

src/AssemblyVersioning.cs

@@ -24,5 +24,5 @@
 //   2) Update the installer version to match the AssemblyVersion below.
 //   These can be found in SqlBuildManager.Setup -> Organize Your Setup -> General Information
 
-[assembly: AssemblyVersion("15.4.2")]
-[assembly: AssemblyFileVersion("15.4.2")]
+[assembly: AssemblyVersion("15.5.0")]
+[assembly: AssemblyFileVersion("15.5.0")]

src/Dockerfile

@@ -1,9 +1,9 @@
 #See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
 
-FROM mcr.microsoft.com/dotnet/runtime:7.0.5-jammy-amd64 AS base
+FROM mcr.microsoft.com/dotnet/runtime:7.0 AS base
 WORKDIR /app
 
-FROM mcr.microsoft.com/dotnet/sdk:7.0.203-jammy-amd64 AS build
+FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
 WORKDIR /src
 
 

src/SqlBuildManager.Console.Dependent.UnitTest/SqlBuildManager.Console.Dependent.UnitTest.csproj

@@ -62,9 +62,9 @@
 
     <PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
 
-    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.6.3" />
-    <PackageReference Include="MSTest.TestAdapter" Version="3.0.4" />
-    <PackageReference Include="MSTest.TestFramework" Version="3.0.4" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
+    <PackageReference Include="MSTest.TestAdapter" Version="3.1.1" />
+    <PackageReference Include="MSTest.TestFramework" Version="3.1.1" />
     <PackageReference Include="coverlet.collector" Version="6.0.0">
       <PrivateAssets>all</PrivateAssets>
       <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>