From 4ce3086256f94696f96adfb6c7c41f46c3a5ab5e Mon Sep 17 00:00:00 2001 From: Michael McKechney Date: Tue, 11 Apr 2023 15:42:58 -0400 Subject: [PATCH] v15.3.0 - see changelog.md --- CHANGELOG.md | 44 +++++++++++++- src/AssemblyVersioning.cs | 4 +- src/Dockerfile | 4 +- .../BatchTests.cs | 27 +++++++-- ...qlBuildManager.Console.ExternalTest.csproj | 2 +- .../CommandLine/CommandLineBuilder.Aci.cs | 1 - .../Worker/Worker.Batch.cs | 2 +- src/SqlBuildManager.Console/sbm.csproj | 16 ++--- .../SqlBuildManager.Logging.csproj | 2 +- .../SqlBuildManager.SetUp.vdproj | 58 ++++++++++++++++--- .../SqlSync.Connection.csproj | 2 +- .../SqlSync.SprocTest.csproj | 2 +- ...SqlSync.SqlBuild.Dependent.UnitTest.csproj | 2 +- .../SqlSync.SqlBuild.UnitTest.csproj | 2 +- src/SqlSync.SqlBuild/SQLSync.SqlBuild.csproj | 2 +- src/SqlSync.SqlBuild/SqlSync.SqlBuild.csproj | 2 +- .../SqlSync.TableScript.csproj | 2 +- src/SqlSync/SQLSync.csproj | 2 +- 18 files changed, 138 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 80e46168..74e94cdc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,52 @@ # SQL Build Manager Change Log +### Version 15.3.0 +_Consolidated updates in Version 15+_ +*NEW:* Removing `beta` tag as the new AKS [Workload Identity](https://docs.microsoft.com/en-us/azure/aks/use-managed-identity) implementation is now GA. This replaces of AAD Pod Identity and is a *breaking change* from any previous Kubernetes deployments. To understand how to configure your cluster, review the steps in the [create_aks_cluster.ps1](scripts/templates/kubernetes/create_aks_cluster.ps1) script. + +_New & Updated Commands:_ +- Eliminated the need for `sbm batch enqueue`. You can now run `sbm batch run` and it will automatically enqueue the database targets for you. You can still run `sbm batch enqueue` first if desired +- `sbm utilty override` command to generate an override cfg file from a SQL script file. +- `sbm k8s query` command to run a query across your database fleet using Kubernetes as a compute platform +- `sbm aci run` command to orchrstrate full ACI process (prep, enqueue, deploy and monitor commands) +- `sbm aci query` command to run a query across your database fleet using ACI as a compute platform +- `sbm batch query` command now fully supports reading messages from Service Bus as well as using Managed Identity + + +_New Options:_ +- `--tenantid` option to provide Azure AD Tenant ID for deployments. This will be necessary if local ID has access to multiple tenants and the target tenant is not the default +- `--batchresourcegroup` (`--batchrg`) argument to specify the resource group for the Batch account. (If not provided, will be infered from Identity resource group) +- `--podcount` for Kubernetes deployments to specify the number of pods to deploy per job +- `--vnetresourcegroup` (`--vnetrg`) argument to specify the resource group for the VNET. (If not provided, will be infered from compute resource group) +- `--eventhublogging` controls how to log script results and if to emit verbose message events. Add multiple flags to combine settings. Values: `EssentialOnly`, `IndividualScriptResults`, `ConsolidatedScriptResults`, `VerboseMessages` + +_New Configuration:_ +- Changed Sample/Test environment to use VNET connections between databases and compute platforms + - SQL Server private VNET connections only, with local firewall rules and excluding "Azure Services" + - VNET integration for Azure Container Apps + - VNET integration for Azure Container Instances + - VNET integration for Batch Nodes + - AKS cluster creation now has Azure RBAC enabled + +_Bug Fixes & Improvements:_ +- EventHub logging now also includes the script results for each script run against the databases as an option. (`--eventhublogging` options of `IndividualScriptResults` or `ConsolidatedScriptResults`) +- Renewing Service Bus message lease every 30 seconds until the build is complete for the target database +- Fixed regression in Batch processing from generated settings files +- Code refactoring for consistency and ease of maintenance +- Corrected bug where Batch execution wasn't properly consolidating certain log files +- *BREAKING CHANGE*: Changed ACI deployment to use SDK vs custom ARM templates. Review new command arguments for `sbm aci prep` and `sbm aci deploy` (and consider using new `sbm aci run` command) + +_Platform updates:_ +- Application now targets .NET 7 +- Docker base images updated to .NET Runtime 7.0.5 and .NET SDK to 7.0.203 +- General code cleanup and switch from System.Data.SqlClient for Microsoft.Data.SqlClient + + ### Version 15.2.2-beta - *NEW:* Eliminated the need for `sbm batch enqueue`. You can now run `sbm batch run` and it will automatically enqueue the database targets for you. You can still run `sbm batch enqueue` first if desired - *ADDED:* Renewing Service Bus message lease every 30 seconds until the build is complete for the target database - *ADDED:* EventHub logging now also includes the script results for each script run against the databases. -- *ADDED:* New option `--eventhublogging` controls how to log script results and if to emit verbose message events. Add multiple flags to combine settings. Values: EssentialOnly|IndividualScriptResults|ConsolidatedScriptResults|VerboseMessages") +- *ADDED:* New option `--eventhublogging` controls how to log script results and if to emit verbose message events. Add multiple flags to combine settings. Values: `EssentialOnly`, `IndividualScriptResults`, `ConsolidatedScriptResults`, `VerboseMessages` - *FIXED:* Corrected bug where Batch execution wasn't properly consolidating certain log files ### Version 15.2.1-beta diff --git a/src/AssemblyVersioning.cs b/src/AssemblyVersioning.cs index 7af202d7..cae3f26c 100644 --- a/src/AssemblyVersioning.cs +++ b/src/AssemblyVersioning.cs @@ -24,5 +24,5 @@ // 2) Update the installer version to match the AssemblyVersion below. // These can be found in SqlBuildManager.Setup -> Organize Your Setup -> General Information -[assembly: AssemblyVersion("15.2.2")] -[assembly: AssemblyFileVersion("15.2.2")] +[assembly: AssemblyVersion("15.3.0")] +[assembly: AssemblyFileVersion("15.3.0")] diff --git a/src/Dockerfile b/src/Dockerfile index c9f0c6fa..44f6dd15 100644 --- a/src/Dockerfile +++ b/src/Dockerfile @@ -1,9 +1,9 @@ #See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging. -FROM mcr.microsoft.com/dotnet/runtime:7.0.2-jammy-amd64 AS base +FROM mcr.microsoft.com/dotnet/runtime:7.0.5-jammy-amd64 AS base WORKDIR /app -FROM mcr.microsoft.com/dotnet/sdk:7.0.102-jammy-amd64 AS build +FROM mcr.microsoft.com/dotnet/sdk:7.0.203-jammy-amd64 AS build WORKDIR /src diff --git a/src/SqlBuildManager.Console.ExternalTest/BatchTests.cs b/src/SqlBuildManager.Console.ExternalTest/BatchTests.cs index 22fb2928..b04d3e2e 100644 --- a/src/SqlBuildManager.Console.ExternalTest/BatchTests.cs +++ b/src/SqlBuildManager.Console.ExternalTest/BatchTests.cs @@ -1579,9 +1579,26 @@ public void Batch_Queue_LongRunning_SBMSource_ByConcurrencyType_Success(string b int startingLine = LogFileCurrentLineCount(); var tmpOverride = Path.Combine(Path.GetDirectoryName(overrideFilePath), Guid.NewGuid().ToString() + ".cfg"); File.WriteAllLines(tmpOverride, overrideFileContents.Take(3).ToList().ToArray()); - var args = new string[]{ + "batch", "enqueue", + "--settingsfile", settingsFile, + "--settingsfilekey", settingsFileKeyPath, + "--override" , tmpOverride, + "--concurrencytype", concurType.ToString(), + "--jobname", jobName}; + + RootCommand rootCommand = CommandLineBuilder.SetUp(); + var val = rootCommand.InvokeAsync(args); + val.Wait(); + var result = val.Result; + + var logFileContents = ReleventLogFileContents(startingLine); + Assert.AreEqual(0, result, StandardExecutionErrorMessage(logFileContents)); + + + + args = new string[]{ "--loglevel", "debug", "batch", batchMethod, "--settingsfile", settingsFile, @@ -1596,12 +1613,12 @@ public void Batch_Queue_LongRunning_SBMSource_ByConcurrencyType_Success(string b "--stream", "--eventhublogging", EventHubLogging.IndividualScriptResults.ToString()}; - RootCommand rootCommand = CommandLineBuilder.SetUp(); - Task val = rootCommand.InvokeAsync(args); + rootCommand = CommandLineBuilder.SetUp(); + val = rootCommand.InvokeAsync(args); val.Wait(); - var result = val.Result; + result = val.Result; - var logFileContents = ReleventLogFileContents(startingLine); + logFileContents = ReleventLogFileContents(startingLine); Assert.AreEqual(0, result, StandardExecutionErrorMessage(logFileContents)); } diff --git a/src/SqlBuildManager.Console.ExternalTest/SqlBuildManager.Console.ExternalTest.csproj b/src/SqlBuildManager.Console.ExternalTest/SqlBuildManager.Console.ExternalTest.csproj index 223346e1..1127ca47 100644 --- a/src/SqlBuildManager.Console.ExternalTest/SqlBuildManager.Console.ExternalTest.csproj +++ b/src/SqlBuildManager.Console.ExternalTest/SqlBuildManager.Console.ExternalTest.csproj @@ -16,7 +16,7 @@ runtime; build; native; contentfiles; analyzers; buildtransitive - + diff --git a/src/SqlBuildManager.Console/CommandLine/CommandLineBuilder.Aci.cs b/src/SqlBuildManager.Console/CommandLine/CommandLineBuilder.Aci.cs index 0dfbeab3..f5c8c3c9 100644 --- a/src/SqlBuildManager.Console/CommandLine/CommandLineBuilder.Aci.cs +++ b/src/SqlBuildManager.Console/CommandLine/CommandLineBuilder.Aci.cs @@ -205,7 +205,6 @@ private static Command AciDeployCommand eventHubLoggingTypeOption, identityNameOption.Copy(false), identityResourceGroupOption.Copy(false), - eventHubLoggingTypeOption, sectionPlaceholderOption, jobnameOption.Copy(true), packagenameAsFileToUploadOption, diff --git a/src/SqlBuildManager.Console/Worker/Worker.Batch.cs b/src/SqlBuildManager.Console/Worker/Worker.Batch.cs index a7ac7694..4f3e2a7c 100644 --- a/src/SqlBuildManager.Console/Worker/Worker.Batch.cs +++ b/src/SqlBuildManager.Console/Worker/Worker.Batch.cs @@ -178,7 +178,7 @@ internal async static Task Batch_RunBuild(CommandLineArgs cmdLine, bool mon private static Task batchMonitorTask = null; private static void Batch_MonitorStart(object sender, BatchMonitorEventArgs e) { - batchMonitorTask = MonitorServiceBusRuntimeProgress(e.CmdLine, e.Stream, DateTime.UtcNow, e.UnitTest); + batchMonitorTask = MonitorServiceBusRuntimeProgress(e.CmdLine, e.Stream, DateTime.UtcNow, e.UnitTest, false); } private static void Batch_MonitorEnd(object sender, EventArgs e) { diff --git a/src/SqlBuildManager.Console/sbm.csproj b/src/SqlBuildManager.Console/sbm.csproj index 91809369..a53b0c4e 100644 --- a/src/SqlBuildManager.Console/sbm.csproj +++ b/src/SqlBuildManager.Console/sbm.csproj @@ -23,14 +23,14 @@ - - + + - - + + - + @@ -42,7 +42,7 @@ - + @@ -60,8 +60,8 @@ - - + + diff --git a/src/SqlBuildManager.Logging/SqlBuildManager.Logging.csproj b/src/SqlBuildManager.Logging/SqlBuildManager.Logging.csproj index 49ba5e7a..36171b59 100644 --- a/src/SqlBuildManager.Logging/SqlBuildManager.Logging.csproj +++ b/src/SqlBuildManager.Logging/SqlBuildManager.Logging.csproj @@ -15,7 +15,7 @@ - + diff --git a/src/SqlBuildManager.SetUp/SqlBuildManager.SetUp.vdproj b/src/SqlBuildManager.SetUp/SqlBuildManager.SetUp.vdproj index 461dbdac..18606412 100644 --- a/src/SqlBuildManager.SetUp/SqlBuildManager.SetUp.vdproj +++ b/src/SqlBuildManager.SetUp/SqlBuildManager.SetUp.vdproj @@ -429,6 +429,12 @@ } "Entry" { + "MsmKey" = "8:_EA2969DD63AF407B80F2832C511ABFC4" + "OwnerKey" = "8:_UNDEFINED" + "MsmSig" = "8:_UNDEFINED" + } + "Entry" + { "MsmKey" = "8:_F3BA57C1CE1048A38B511DC7CACBE960" "OwnerKey" = "8:_UNDEFINED" "MsmSig" = "8:_UNDEFINED" @@ -482,6 +488,14 @@ "PrerequisitesLocation" = "2:1" "Url" = "8:" "ComponentsUrl" = "8:" + "Items" + { + "{EDC2488A-8267-493A-A98E-7D9C3B36CDF3}:.NETFramework,Version=v4.7.2" + { + "Name" = "8:Microsoft .NET Framework 4.7.2 (x86 and x64)" + "ProductCode" = "8:.NETFramework,Version=v4.7.2" + } + } } } "Release" @@ -506,6 +520,14 @@ "PrerequisitesLocation" = "2:1" "Url" = "8:" "ComponentsUrl" = "8:" + "Items" + { + "{EDC2488A-8267-493A-A98E-7D9C3B36CDF3}:.NETFramework,Version=v4.7.2" + { + "Name" = "8:Microsoft .NET Framework 4.7.2 (x86 and x64)" + "ProductCode" = "8:.NETFramework,Version=v4.7.2" + } + } } } } @@ -1858,6 +1880,26 @@ "IsDependency" = "11:FALSE" "IsolateTo" = "8:" } + "{1FB2D0AE-D3B9-43D4-B9DD-F88EC61E35DE}:_EA2969DD63AF407B80F2832C511ABFC4" + { + "SourcePath" = "8:..\\SqlSync\\bin\\Release\\net7.0-windows\\SqlBuildManager.exe" + "TargetName" = "8:SqlBuildManager.exe" + "Tag" = "8:" + "Folder" = "8:_39C777AAEF4746B49C67DCAA99BC024F" + "Condition" = "8:" + "Transitive" = "11:FALSE" + "Vital" = "11:TRUE" + "ReadOnly" = "11:FALSE" + "Hidden" = "11:FALSE" + "System" = "11:FALSE" + "Permanent" = "11:FALSE" + "SharedLegacy" = "11:FALSE" + "PackageAs" = "3:1" + "Register" = "3:1" + "Exclude" = "11:FALSE" + "IsDependency" = "11:FALSE" + "IsolateTo" = "8:" + } "{1FB2D0AE-D3B9-43D4-B9DD-F88EC61E35DE}:_F3BA57C1CE1048A38B511DC7CACBE960" { "SourcePath" = "8:..\\SqlSync\\SqlBuild\\Utility\\Wrap Add Index.sql" @@ -2346,15 +2388,15 @@ { "Name" = "8:Microsoft Visual Studio" "ProductName" = "8:SQL Build Manager" - "ProductCode" = "8:{2DB34965-7692-4C0F-853F-53AA1C74BA87}" - "PackageCode" = "8:{88BA03E3-4687-4E04-A3BC-EFF785C7FBAC}" + "ProductCode" = "8:{FD6BDF72-A510-4B26-9F0E-53AB44C18980}" + "PackageCode" = "8:{D4DF09A2-5CDD-496C-894D-656ED9F3DC1A}" "UpgradeCode" = "8:{00F0500D-7970-463C-9EBE-5BEB116F0DAA}" "AspNetVersion" = "8:4.0.30319.0" "RestartWWWService" = "11:FALSE" "RemovePreviousVersions" = "11:FALSE" "DetectNewerInstalledVersion" = "11:TRUE" "InstallAllUsers" = "11:FALSE" - "ProductVersion" = "8:14.6.0" + "ProductVersion" = "8:15.3.0" "Manufacturer" = "8:BlueSkyDevUs" "ARPHELPTELEPHONE" = "8:" "ARPHELPLINK" = "8:http://blueskydev.us" @@ -2475,7 +2517,7 @@ "ShowCmd" = "3:1" "IconIndex" = "3:0" "Transitive" = "11:FALSE" - "Target" = "8:_1E0A882B2ABC473CB3CE87DF188C0ADD" + "Target" = "8:_EA2969DD63AF407B80F2832C511ABFC4" "Folder" = "8:_77FACB82935C44238697948FE5DA8673" "WorkingFolder" = "8:_39C777AAEF4746B49C67DCAA99BC024F" "Icon" = "8:_37CC4BB593BC4B78AF85FDF86B1EBE14" @@ -2489,7 +2531,7 @@ "ShowCmd" = "3:1" "IconIndex" = "3:0" "Transitive" = "11:FALSE" - "Target" = "8:_1E0A882B2ABC473CB3CE87DF188C0ADD" + "Target" = "8:_EA2969DD63AF407B80F2832C511ABFC4" "Folder" = "8:_2A2FFD2F59F44804AF8540F849561C56" "WorkingFolder" = "8:_39C777AAEF4746B49C67DCAA99BC024F" "Icon" = "8:_37CC4BB593BC4B78AF85FDF86B1EBE14" @@ -2901,7 +2943,7 @@ { "{5259A561-127C-4D43-A0A1-72F10C7B3BF8}:_1E0A882B2ABC473CB3CE87DF188C0ADD" { - "SourcePath" = "8:..\\SqlSync\\obj\\Release\\net6.0-windows\\SqlBuildManager.dll" + "SourcePath" = "8:..\\SqlSync\\obj\\Release\\net7.0-windows\\SqlBuildManager.dll" "TargetName" = "8:" "Tag" = "8:" "Folder" = "8:_39C777AAEF4746B49C67DCAA99BC024F" @@ -2985,7 +3027,7 @@ } "{5259A561-127C-4D43-A0A1-72F10C7B3BF8}:_875DDF1B8A8D4B4AA4DE8B65F349DDBE" { - "SourcePath" = "8:..\\SqlBuildManager.Console\\obj\\Release\\net6.0\\apphost.exe" + "SourcePath" = "8:..\\SqlBuildManager.Console\\obj\\Release\\net7.0\\apphost.exe" "TargetName" = "8:" "Tag" = "8:" "Folder" = "8:_39C777AAEF4746B49C67DCAA99BC024F" @@ -3069,7 +3111,7 @@ } "{5259A561-127C-4D43-A0A1-72F10C7B3BF8}:_C3CBF3D7F53E4B5E838ADB6050E29E56" { - "SourcePath" = "8:..\\SqlSync\\obj\\Release\\net6.0-windows\\apphost.exe" + "SourcePath" = "8:..\\SqlSync\\obj\\Release\\net7.0-windows\\apphost.exe" "TargetName" = "8:" "Tag" = "8:" "Folder" = "8:_39C777AAEF4746B49C67DCAA99BC024F" diff --git a/src/SqlSync.Connection/SqlSync.Connection.csproj b/src/SqlSync.Connection/SqlSync.Connection.csproj index ca777b03..cc5b0dff 100644 --- a/src/SqlSync.Connection/SqlSync.Connection.csproj +++ b/src/SqlSync.Connection/SqlSync.Connection.csproj @@ -6,7 +6,7 @@ - + diff --git a/src/SqlSync.SprocTest/SqlSync.SprocTest.csproj b/src/SqlSync.SprocTest/SqlSync.SprocTest.csproj index caf348c3..ebe29207 100644 --- a/src/SqlSync.SprocTest/SqlSync.SprocTest.csproj +++ b/src/SqlSync.SprocTest/SqlSync.SprocTest.csproj @@ -6,7 +6,7 @@ false - + diff --git a/src/SqlSync.SqlBuild.Dependent.UnitTest/SqlSync.SqlBuild.Dependent.UnitTest.csproj b/src/SqlSync.SqlBuild.Dependent.UnitTest/SqlSync.SqlBuild.Dependent.UnitTest.csproj index 2917a15e..0f57f4bc 100644 --- a/src/SqlSync.SqlBuild.Dependent.UnitTest/SqlSync.SqlBuild.Dependent.UnitTest.csproj +++ b/src/SqlSync.SqlBuild.Dependent.UnitTest/SqlSync.SqlBuild.Dependent.UnitTest.csproj @@ -31,7 +31,7 @@ - + diff --git a/src/SqlSync.SqlBuild.UnitTest/SqlSync.SqlBuild.UnitTest.csproj b/src/SqlSync.SqlBuild.UnitTest/SqlSync.SqlBuild.UnitTest.csproj index 84634bcc..fa695e21 100644 --- a/src/SqlSync.SqlBuild.UnitTest/SqlSync.SqlBuild.UnitTest.csproj +++ b/src/SqlSync.SqlBuild.UnitTest/SqlSync.SqlBuild.UnitTest.csproj @@ -20,7 +20,7 @@ - + diff --git a/src/SqlSync.SqlBuild/SQLSync.SqlBuild.csproj b/src/SqlSync.SqlBuild/SQLSync.SqlBuild.csproj index 0ad4ea92..440eeb2d 100644 --- a/src/SqlSync.SqlBuild/SQLSync.SqlBuild.csproj +++ b/src/SqlSync.SqlBuild/SQLSync.SqlBuild.csproj @@ -16,7 +16,7 @@ - + diff --git a/src/SqlSync.SqlBuild/SqlSync.SqlBuild.csproj b/src/SqlSync.SqlBuild/SqlSync.SqlBuild.csproj index 0ad4ea92..440eeb2d 100644 --- a/src/SqlSync.SqlBuild/SqlSync.SqlBuild.csproj +++ b/src/SqlSync.SqlBuild/SqlSync.SqlBuild.csproj @@ -16,7 +16,7 @@ - + diff --git a/src/SqlSync.TableScript/SqlSync.TableScript.csproj b/src/SqlSync.TableScript/SqlSync.TableScript.csproj index f52cb43a..af3228f1 100644 --- a/src/SqlSync.TableScript/SqlSync.TableScript.csproj +++ b/src/SqlSync.TableScript/SqlSync.TableScript.csproj @@ -49,7 +49,7 @@ - + diff --git a/src/SqlSync/SQLSync.csproj b/src/SqlSync/SQLSync.csproj index ce9f524b..7829cbec 100644 --- a/src/SqlSync/SQLSync.csproj +++ b/src/SqlSync/SQLSync.csproj @@ -345,7 +345,7 @@ - +