Skip to content

Commit

Permalink
v15.3.0 - see changelog.md
Browse files Browse the repository at this point in the history
  • Loading branch information
mmckechney committed Apr 28, 2023
1 parent 8c6e72a commit 4ce3086
Show file tree
Hide file tree
Showing 18 changed files with 138 additions and 38 deletions.
44 changes: 43 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,52 @@
# SQL Build Manager Change Log

### Version 15.3.0
_Consolidated updates in Version 15+_
*NEW:* Removing `beta` tag as the new AKS [Workload Identity](https://docs.microsoft.com/en-us/azure/aks/use-managed-identity) implementation is now GA. This replaces of AAD Pod Identity and is a *breaking change* from any previous Kubernetes deployments. To understand how to configure your cluster, review the steps in the [create_aks_cluster.ps1](scripts/templates/kubernetes/create_aks_cluster.ps1) script.

_New & Updated Commands:_
- Eliminated the need for `sbm batch enqueue`. You can now run `sbm batch run` and it will automatically enqueue the database targets for you. You can still run `sbm batch enqueue` first if desired
- `sbm utilty override` command to generate an override cfg file from a SQL script file.
- `sbm k8s query` command to run a query across your database fleet using Kubernetes as a compute platform
- `sbm aci run` command to orchrstrate full ACI process (prep, enqueue, deploy and monitor commands)
- `sbm aci query` command to run a query across your database fleet using ACI as a compute platform
- `sbm batch query` command now fully supports reading messages from Service Bus as well as using Managed Identity


_New Options:_
- `--tenantid` option to provide Azure AD Tenant ID for deployments. This will be necessary if local ID has access to multiple tenants and the target tenant is not the default
- `--batchresourcegroup` (`--batchrg`) argument to specify the resource group for the Batch account. (If not provided, will be infered from Identity resource group)
- `--podcount` for Kubernetes deployments to specify the number of pods to deploy per job
- `--vnetresourcegroup` (`--vnetrg`) argument to specify the resource group for the VNET. (If not provided, will be infered from compute resource group)
- `--eventhublogging` controls how to log script results and if to emit verbose message events. Add multiple flags to combine settings. Values: `EssentialOnly`, `IndividualScriptResults`, `ConsolidatedScriptResults`, `VerboseMessages`

_New Configuration:_
- Changed Sample/Test environment to use VNET connections between databases and compute platforms
- SQL Server private VNET connections only, with local firewall rules and excluding "Azure Services"
- VNET integration for Azure Container Apps
- VNET integration for Azure Container Instances
- VNET integration for Batch Nodes
- AKS cluster creation now has Azure RBAC enabled

_Bug Fixes & Improvements:_
- EventHub logging now also includes the script results for each script run against the databases as an option. (`--eventhublogging` options of `IndividualScriptResults` or `ConsolidatedScriptResults`)
- Renewing Service Bus message lease every 30 seconds until the build is complete for the target database
- Fixed regression in Batch processing from generated settings files
- Code refactoring for consistency and ease of maintenance
- Corrected bug where Batch execution wasn't properly consolidating certain log files
- *BREAKING CHANGE*: Changed ACI deployment to use SDK vs custom ARM templates. Review new command arguments for `sbm aci prep` and `sbm aci deploy` (and consider using new `sbm aci run` command)

_Platform updates:_
- Application now targets .NET 7
- Docker base images updated to .NET Runtime 7.0.5 and .NET SDK to 7.0.203
- General code cleanup and switch from System.Data.SqlClient for Microsoft.Data.SqlClient


### Version 15.2.2-beta
- *NEW:* Eliminated the need for `sbm batch enqueue`. You can now run `sbm batch run` and it will automatically enqueue the database targets for you. You can still run `sbm batch enqueue` first if desired
- *ADDED:* Renewing Service Bus message lease every 30 seconds until the build is complete for the target database
- *ADDED:* EventHub logging now also includes the script results for each script run against the databases.
- *ADDED:* New option `--eventhublogging` controls how to log script results and if to emit verbose message events. Add multiple flags to combine settings. Values: EssentialOnly|IndividualScriptResults|ConsolidatedScriptResults|VerboseMessages")
- *ADDED:* New option `--eventhublogging` controls how to log script results and if to emit verbose message events. Add multiple flags to combine settings. Values: `EssentialOnly`, `IndividualScriptResults`, `ConsolidatedScriptResults`, `VerboseMessages`
- *FIXED:* Corrected bug where Batch execution wasn't properly consolidating certain log files

### Version 15.2.1-beta
Expand Down
4 changes: 2 additions & 2 deletions src/AssemblyVersioning.cs
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,5 @@
// 2) Update the installer version to match the AssemblyVersion below.
// These can be found in SqlBuildManager.Setup -> Organize Your Setup -> General Information

[assembly: AssemblyVersion("15.2.2")]
[assembly: AssemblyFileVersion("15.2.2")]
[assembly: AssemblyVersion("15.3.0")]
[assembly: AssemblyFileVersion("15.3.0")]
4 changes: 2 additions & 2 deletions src/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.

FROM mcr.microsoft.com/dotnet/runtime:7.0.2-jammy-amd64 AS base
FROM mcr.microsoft.com/dotnet/runtime:7.0.5-jammy-amd64 AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:7.0.102-jammy-amd64 AS build
FROM mcr.microsoft.com/dotnet/sdk:7.0.203-jammy-amd64 AS build
WORKDIR /src


Expand Down
27 changes: 22 additions & 5 deletions src/SqlBuildManager.Console.ExternalTest/BatchTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1579,9 +1579,26 @@ public void Batch_Queue_LongRunning_SBMSource_ByConcurrencyType_Success(string b
int startingLine = LogFileCurrentLineCount();
var tmpOverride = Path.Combine(Path.GetDirectoryName(overrideFilePath), Guid.NewGuid().ToString() + ".cfg");
File.WriteAllLines(tmpOverride, overrideFileContents.Take(3).ToList().ToArray());


var args = new string[]{
"batch", "enqueue",
"--settingsfile", settingsFile,
"--settingsfilekey", settingsFileKeyPath,
"--override" , tmpOverride,
"--concurrencytype", concurType.ToString(),
"--jobname", jobName};

RootCommand rootCommand = CommandLineBuilder.SetUp();
var val = rootCommand.InvokeAsync(args);
val.Wait();
var result = val.Result;

var logFileContents = ReleventLogFileContents(startingLine);
Assert.AreEqual(0, result, StandardExecutionErrorMessage(logFileContents));



args = new string[]{
"--loglevel", "debug",
"batch", batchMethod,
"--settingsfile", settingsFile,
Expand All @@ -1596,12 +1613,12 @@ public void Batch_Queue_LongRunning_SBMSource_ByConcurrencyType_Success(string b
"--stream",
"--eventhublogging", EventHubLogging.IndividualScriptResults.ToString()};

RootCommand rootCommand = CommandLineBuilder.SetUp();
Task<int> val = rootCommand.InvokeAsync(args);
rootCommand = CommandLineBuilder.SetUp();
val = rootCommand.InvokeAsync(args);
val.Wait();
var result = val.Result;
result = val.Result;

var logFileContents = ReleventLogFileContents(startingLine);
logFileContents = ReleventLogFileContents(startingLine);
Assert.AreEqual(0, result, StandardExecutionErrorMessage(logFileContents));
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="System.CommandLine" Version="2.0.0-beta4.22272.1" />
<PackageReference Include="YamlDotNet" Version="13.0.2" />
<PackageReference Include="YamlDotNet" Version="13.1.0" />
</ItemGroup>
<ItemGroup>
<!--<ProjectReference Include="..\..\..\command-line-api\src\System.CommandLine\System.CommandLine.csproj" />-->
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,6 @@ private static Command AciDeployCommand
eventHubLoggingTypeOption,
identityNameOption.Copy(false),
identityResourceGroupOption.Copy(false),
eventHubLoggingTypeOption,
sectionPlaceholderOption,
jobnameOption.Copy(true),
packagenameAsFileToUploadOption,
Expand Down
2 changes: 1 addition & 1 deletion src/SqlBuildManager.Console/Worker/Worker.Batch.cs
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ internal async static Task<int> Batch_RunBuild(CommandLineArgs cmdLine, bool mon
private static Task<int> batchMonitorTask = null;
private static void Batch_MonitorStart(object sender, BatchMonitorEventArgs e)
{
batchMonitorTask = MonitorServiceBusRuntimeProgress(e.CmdLine, e.Stream, DateTime.UtcNow, e.UnitTest);
batchMonitorTask = MonitorServiceBusRuntimeProgress(e.CmdLine, e.Stream, DateTime.UtcNow, e.UnitTest, false);
}
private static void Batch_MonitorEnd(object sender, EventArgs e)
{
Expand Down
16 changes: 8 additions & 8 deletions src/SqlBuildManager.Console/sbm.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Azure.Identity" Version="1.8.2" />
<PackageReference Include="Azure.Messaging.EventHubs" Version="5.8.1" />
<PackageReference Include="Azure.Messaging.EventHubs.Processor" Version="5.8.1" />
<PackageReference Include="Azure.Messaging.EventHubs" Version="5.9.0" />
<PackageReference Include="Azure.Messaging.EventHubs.Processor" Version="5.9.0" />
<PackageReference Include="Azure.Messaging.ServiceBus" Version="7.13.1" />
<PackageReference Include="Azure.ResourceManager" Version="1.4.0" />
<PackageReference Include="Azure.ResourceManager.AppContainers" Version="1.0.2" />
<PackageReference Include="Azure.ResourceManager" Version="1.5.0" />
<PackageReference Include="Azure.ResourceManager.AppContainers" Version="1.0.3" />
<PackageReference Include="Azure.ResourceManager.Batch" Version="1.1.1" />
<PackageReference Include="Azure.ResourceManager.ContainerInstance" Version="1.0.1" />
<PackageReference Include="Azure.ResourceManager.Network" Version="1.1.1" />
<PackageReference Include="Azure.ResourceManager.Network" Version="1.2.0" />
<PackageReference Include="Azure.Security.KeyVault.Secrets" Version="4.5.0" />
<PackageReference Include="Azure.ResourceManager.Resources" Version="1.4.0" />
<PackageReference Include="Microsoft.Azure.Management.Batch" Version="15.0.0" />
Expand All @@ -42,7 +42,7 @@
<PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="7.0.0" />
<PackageReference Include="Microsoft.SqlServer.DacFx" Version="161.8089.0" />
<PackageReference Include="morelinq" Version="3.4.1" />
<PackageReference Include="morelinq" Version="3.4.2" />
<PackageReference Include="Serilog.Enrichers.Thread" Version="3.1.0" />
<PackageReference Include="Serilog.Expressions" Version="3.4.1" />
<PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
Expand All @@ -60,8 +60,8 @@
<PackageReference Include="Microsoft.Azure.Batch" Version="15.4.0" />
<PackageReference Include="Microsoft.Rest.ClientRuntime" Version="3.0.3" />
<PackageReference Include="Microsoft.Rest.ClientRuntime.Azure" Version="4.0.3" />
<PackageReference Include="Azure.Storage.Blobs" Version="12.15.0" />
<PackageReference Include="YamlDotNet" Version="13.0.2" />
<PackageReference Include="Azure.Storage.Blobs" Version="12.16.0" />
<PackageReference Include="YamlDotNet" Version="13.1.0" />
</ItemGroup>
<ItemGroup>
<None Update="Aci\aci_arm_template.json">
Expand Down
2 changes: 1 addition & 1 deletion src/SqlBuildManager.Logging/SqlBuildManager.Logging.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

<ItemGroup>
<PackageReference Include="Azure.Identity" Version="1.8.2" />
<PackageReference Include="Azure.Messaging.EventHubs" Version="5.8.1" />
<PackageReference Include="Azure.Messaging.EventHubs" Version="5.9.0" />
<PackageReference Include="Serilog.Enrichers.Thread" Version="3.1.0" />
<PackageReference Include="Serilog.Expressions" Version="3.4.1" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.4.0" />
Expand Down
58 changes: 50 additions & 8 deletions src/SqlBuildManager.SetUp/SqlBuildManager.SetUp.vdproj
Original file line number Diff line number Diff line change
Expand Up @@ -429,6 +429,12 @@
}
"Entry"
{
"MsmKey" = "8:_EA2969DD63AF407B80F2832C511ABFC4"
"OwnerKey" = "8:_UNDEFINED"
"MsmSig" = "8:_UNDEFINED"
}
"Entry"
{
"MsmKey" = "8:_F3BA57C1CE1048A38B511DC7CACBE960"
"OwnerKey" = "8:_UNDEFINED"
"MsmSig" = "8:_UNDEFINED"
Expand Down Expand Up @@ -482,6 +488,14 @@
"PrerequisitesLocation" = "2:1"
"Url" = "8:"
"ComponentsUrl" = "8:"
"Items"
{
"{EDC2488A-8267-493A-A98E-7D9C3B36CDF3}:.NETFramework,Version=v4.7.2"
{
"Name" = "8:Microsoft .NET Framework 4.7.2 (x86 and x64)"
"ProductCode" = "8:.NETFramework,Version=v4.7.2"
}
}
}
}
"Release"
Expand All @@ -506,6 +520,14 @@
"PrerequisitesLocation" = "2:1"
"Url" = "8:"
"ComponentsUrl" = "8:"
"Items"
{
"{EDC2488A-8267-493A-A98E-7D9C3B36CDF3}:.NETFramework,Version=v4.7.2"
{
"Name" = "8:Microsoft .NET Framework 4.7.2 (x86 and x64)"
"ProductCode" = "8:.NETFramework,Version=v4.7.2"
}
}
}
}
}
Expand Down Expand Up @@ -1858,6 +1880,26 @@
"IsDependency" = "11:FALSE"
"IsolateTo" = "8:"
}
"{1FB2D0AE-D3B9-43D4-B9DD-F88EC61E35DE}:_EA2969DD63AF407B80F2832C511ABFC4"
{
"SourcePath" = "8:..\\SqlSync\\bin\\Release\\net7.0-windows\\SqlBuildManager.exe"
"TargetName" = "8:SqlBuildManager.exe"
"Tag" = "8:"
"Folder" = "8:_39C777AAEF4746B49C67DCAA99BC024F"
"Condition" = "8:"
"Transitive" = "11:FALSE"
"Vital" = "11:TRUE"
"ReadOnly" = "11:FALSE"
"Hidden" = "11:FALSE"
"System" = "11:FALSE"
"Permanent" = "11:FALSE"
"SharedLegacy" = "11:FALSE"
"PackageAs" = "3:1"
"Register" = "3:1"
"Exclude" = "11:FALSE"
"IsDependency" = "11:FALSE"
"IsolateTo" = "8:"
}
"{1FB2D0AE-D3B9-43D4-B9DD-F88EC61E35DE}:_F3BA57C1CE1048A38B511DC7CACBE960"
{
"SourcePath" = "8:..\\SqlSync\\SqlBuild\\Utility\\Wrap Add Index.sql"
Expand Down Expand Up @@ -2346,15 +2388,15 @@
{
"Name" = "8:Microsoft Visual Studio"
"ProductName" = "8:SQL Build Manager"
"ProductCode" = "8:{2DB34965-7692-4C0F-853F-53AA1C74BA87}"
"PackageCode" = "8:{88BA03E3-4687-4E04-A3BC-EFF785C7FBAC}"
"ProductCode" = "8:{FD6BDF72-A510-4B26-9F0E-53AB44C18980}"
"PackageCode" = "8:{D4DF09A2-5CDD-496C-894D-656ED9F3DC1A}"
"UpgradeCode" = "8:{00F0500D-7970-463C-9EBE-5BEB116F0DAA}"
"AspNetVersion" = "8:4.0.30319.0"
"RestartWWWService" = "11:FALSE"
"RemovePreviousVersions" = "11:FALSE"
"DetectNewerInstalledVersion" = "11:TRUE"
"InstallAllUsers" = "11:FALSE"
"ProductVersion" = "8:14.6.0"
"ProductVersion" = "8:15.3.0"
"Manufacturer" = "8:BlueSkyDevUs"
"ARPHELPTELEPHONE" = "8:"
"ARPHELPLINK" = "8:http://blueskydev.us"
Expand Down Expand Up @@ -2475,7 +2517,7 @@
"ShowCmd" = "3:1"
"IconIndex" = "3:0"
"Transitive" = "11:FALSE"
"Target" = "8:_1E0A882B2ABC473CB3CE87DF188C0ADD"
"Target" = "8:_EA2969DD63AF407B80F2832C511ABFC4"
"Folder" = "8:_77FACB82935C44238697948FE5DA8673"
"WorkingFolder" = "8:_39C777AAEF4746B49C67DCAA99BC024F"
"Icon" = "8:_37CC4BB593BC4B78AF85FDF86B1EBE14"
Expand All @@ -2489,7 +2531,7 @@
"ShowCmd" = "3:1"
"IconIndex" = "3:0"
"Transitive" = "11:FALSE"
"Target" = "8:_1E0A882B2ABC473CB3CE87DF188C0ADD"
"Target" = "8:_EA2969DD63AF407B80F2832C511ABFC4"
"Folder" = "8:_2A2FFD2F59F44804AF8540F849561C56"
"WorkingFolder" = "8:_39C777AAEF4746B49C67DCAA99BC024F"
"Icon" = "8:_37CC4BB593BC4B78AF85FDF86B1EBE14"
Expand Down Expand Up @@ -2901,7 +2943,7 @@
{
"{5259A561-127C-4D43-A0A1-72F10C7B3BF8}:_1E0A882B2ABC473CB3CE87DF188C0ADD"
{
"SourcePath" = "8:..\\SqlSync\\obj\\Release\\net6.0-windows\\SqlBuildManager.dll"
"SourcePath" = "8:..\\SqlSync\\obj\\Release\\net7.0-windows\\SqlBuildManager.dll"
"TargetName" = "8:"
"Tag" = "8:"
"Folder" = "8:_39C777AAEF4746B49C67DCAA99BC024F"
Expand Down Expand Up @@ -2985,7 +3027,7 @@
}
"{5259A561-127C-4D43-A0A1-72F10C7B3BF8}:_875DDF1B8A8D4B4AA4DE8B65F349DDBE"
{
"SourcePath" = "8:..\\SqlBuildManager.Console\\obj\\Release\\net6.0\\apphost.exe"
"SourcePath" = "8:..\\SqlBuildManager.Console\\obj\\Release\\net7.0\\apphost.exe"
"TargetName" = "8:"
"Tag" = "8:"
"Folder" = "8:_39C777AAEF4746B49C67DCAA99BC024F"
Expand Down Expand Up @@ -3069,7 +3111,7 @@
}
"{5259A561-127C-4D43-A0A1-72F10C7B3BF8}:_C3CBF3D7F53E4B5E838ADB6050E29E56"
{
"SourcePath" = "8:..\\SqlSync\\obj\\Release\\net6.0-windows\\apphost.exe"
"SourcePath" = "8:..\\SqlSync\\obj\\Release\\net7.0-windows\\apphost.exe"
"TargetName" = "8:"
"Tag" = "8:"
"Folder" = "8:_39C777AAEF4746B49C67DCAA99BC024F"
Expand Down
2 changes: 1 addition & 1 deletion src/SqlSync.Connection/SqlSync.Connection.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<Compile Include="..\AssemblyVersioning.cs" Link="Properties\AssemblyVersioning.cs" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.0" />
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.1" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
</ItemGroup>
<ItemGroup>
Expand Down
2 changes: 1 addition & 1 deletion src/SqlSync.SprocTest/SqlSync.SprocTest.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<Deterministic>false</Deterministic>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.0" />
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.1" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
<PackageReference Include="MSTest.TestAdapter" Version="3.0.2" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
<EmbeddedResource Include="Resources\XmlWithNoNamespace.txt" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.0" />
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.1" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
<PackageReference Include="MSTest.TestAdapter" Version="3.0.2" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<EmbeddedResource Include="Resources\serialized_multidb_xml.txt" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.0" />
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.1" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
<PackageReference Include="MSTest.TestAdapter" Version="3.0.2" />
Expand Down
2 changes: 1 addition & 1 deletion src/SqlSync.SqlBuild/SQLSync.SqlBuild.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<EmbeddedResource Include="Status\Xml\ServerReport_summary.xslt" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.0" />
<PackageReference Include="Microsoft.Data.SqlClient" Version="5.1.1" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
<PackageReference Include="Microsoft.SqlServer.DacFx" Version="161.8089.0" />
<PackageReference Include="Polly" Version="7.2.3" />
Expand Down
Loading

0 comments on commit 4ce3086

Please sign in to comment.