Dacpac updates and new 'create' command (#124)

* - *FIXED:* Update to Dacpac change scripts to identify new header delimiter - *FIXED:* Issue creating scripts between incompatable SQL Server versions. Will now output a warning and continue to create the scripts with the flag `AllowIncompatiblePlatform=true` - *ADDED:* New command `sbm create` to create a new SBM file from a list of scripts - *UPDATED:* Can now use Windows auth for DACPAC creation - *UPDATED:* updated Nuget packages - Cleaning up compiler warnings * Updating Dac tooling version * Updating MSTest.TestAdaper and MSTest.TestFramework as per microsoft/testfx#871
mmckechney · Jun 16, 2021 · db67c1d · db67c1d
1 parent 0738839
commit db67c1d
Show file tree

Hide file tree

Showing 56 changed files with 428 additions and 210 deletions.
diff --git a/README.md b/README.md
@@ -11,7 +11,9 @@ SQL Build Manager is a multi-faceted tool to allow you to manage the life-cycle
 - [The Basics](#the-basics)
   - [Build package meta-data](#build-package-meta-data)
   - [Creating a build package](#creating-a-build-package)
+  - [Targeting multiple databases](#targeting-multiple-databases)
 - [Running Locally](/docs/local_build.md)
+- [Command Line reference](/docs/commandline.md)
 - [Leverage Azure Batch for massively parallel updates](/docs/azure_batch.md)
 - [Change notes](docs/change_notes.md)
 - For contributors: [Notes on Building and Unit Testing](/docs/setup_azure_environment.md)
@@ -22,36 +24,36 @@ SQL Build Manager is a multi-faceted tool to allow you to manage the life-cycle
 
 ## Key Features
 
-* Packaging of all of your update scripts and runtime meta-data into a single .sbm (zip file) or leverage data-tier application ([DACPAC](https://docs.microsoft.com/en-us/sql/relational-databases/data-tier-applications/deploy-a-data-tier-application)) deployment across your entire database fleet.
-* Single transaction handling. If any one script fails, the entire package is rolled back, leaving the database unchanged.
-* Handle multiple database updates in one package - seamlessly update all your databases with local threading or massively parallel Azure Batch processing.
-* Full script run management. Control the script order, target database, and transaction handling
-* Trial mode - runs scripts to test against database, then rolls back to leave in pristine state.
-* Automatic logging and version tracking of scripts on a per-server/per-database level
-* Full SHA-1 hashing of individual scripts and complete `.sbm` package files to ensure integrity of the scripts
-* Execution of a build package (see below) is recorded in the database for full tracking of update history, script validation and potential rebuilding of packages
-* Massively parallel execution across thousands of databases utilizing local threading or an Azure Batch execution
+- Packaging of all of your update scripts and runtime meta-data into a single .sbm (zip file) or leverage data-tier application ([DACPAC](https://docs.microsoft.com/en-us/sql/relational-databases/data-tier-applications/deploy-a-data-tier-application)) deployment across your entire database fleet.
+- Single transaction handling. If any one script fails, the entire package is rolled back, leaving the database unchanged.
+- Handle multiple database updates in one package - seamlessly update all your databases with local threading or massively parallel Azure Batch processing.
+- Full script run management. Control the script order, target database, and transaction handling
+- Trial mode - runs scripts to test against database, then rolls back to leave in pristine state.
+- Automatic logging and version tracking of scripts on a per-server/per-database level
+- Full SHA-1 hashing of individual scripts and complete `.sbm` package files to ensure integrity of the scripts
+- Execution of a build package (see below) is recorded in the database for full tracking of update history, script validation and potential rebuilding of packages
+- Massively parallel execution across thousands of databases utilizing local threading or an Azure Batch execution
 
 # The Basics
 
 ## Build Package meta-data
 At the core of the process is the "SQL Build Manager Package" file (.sbm extension).  Under the hood, this file is a Zip file that contains the scripts that constitute your "release" along with a configuration file  (SqlSyncBuildProject.xml) that contains meta data on the scripts and execution parameters:
 
-* `FileName`: Self explanatory, the name of the script file
-* `BuildOrder`: The relative order that the scripts in the package will be run
-* `Description`: Optional description about the script
-* `RollBackOnError`: Option on whether or not to roll back the transaction if there is an error running this script (default: `true`)
-* `CausesBuildFailure`: Option on whether or not to roll back the entire build if there is a failure with this script (default `true`)
-* `DateAdded`: Date and time that the script was added to the package
-* `ScriptId`: System generated GUID identifier for the script
-* `Database`: Target database to run the scripts against. (This can be overridden in the case of multiple DB targets)
-* `StripTransactionText`: Script handling to remove any inline transaction statements (default is `true` because you want SQL Build Manager to handle transactions)
-* `AllowMultipleRuns`: Whether or not this script can be run on the same database multiple times (default is `true` and you should always write scripts so this is viable)
-* `AddedBy`: User ID of the user that added the script to the package
-* `ScriptTimeOut`: Timeout setting for the execution of this script
-* `DateModified`: If the script has been modified after being added, this will be populated (otherwise `DateTime.Min`)
-* `ModifiedBy`: If the script has been modified after being added, this will be populated with the user's ID
-* `Tag`: Optional tag for the script
+- `FileName`: Self explanatory, the name of the script file
+- `BuildOrder`: The relative order that the scripts in the package will be run
+- `Description`: Optional description about the script
+- `RollBackOnError`: Option on whether or not to roll back the transaction if there is an error running this script (default: `true`)
+- `CausesBuildFailure`: Option on whether or not to roll back the entire build if there is a failure with this script (default `true`)
+- `DateAdded`: Date and time that the script was added to the package
+- `ScriptId`: System generated GUID identifier for the script
+- `Database`: Target database to run the scripts against. (This can be overridden in the case of multiple DB targets)
+- `StripTransactionText`: Script handling to remove any inline transaction statements (default is `true` because you want SQL Build Manager to handle transactions)
+- `AllowMultipleRuns`: Whether or not this script can be run on the same database multiple times (default is `true` and you should always write scripts so this is viable)
+- `AddedBy`: User ID of the user that added the script to the package
+- `ScriptTimeOut`: Timeout setting for the execution of this script
+- `DateModified`: If the script has been modified after being added, this will be populated (otherwise `DateTime.Min`)
+- `ModifiedBy`: If the script has been modified after being added, this will be populated with the user's ID
+- `Tag`: Optional tag for the script
 
 If you are using a DACPAC deployment, this all gets generated for you based on your command line parameters and defaults
 
@@ -79,9 +81,21 @@ While the focus of the app has changed to command line automation, the forms GUI
 
 ### Command line
 
-The command line utility is geared more toward executing a build vs. creating the package itself. You can however extract a build package file from a DACPAC file using the `sbm scriptextract` command. This is useful if you are utilizing the recommended [data-tier application](https://docs.microsoft.com/en-us/sql/relational-databases/data-tier-applications/data-tier-applications?view=sql-server-2017)  method.
+The command line utility is geared more toward executing a build vs. creating the package itself. You can however create a build package file in a few ways:
 
-A DACPAC can also be created directly from a target "Platinum Database" (why platinum? because it's even more precious than gold!). Using the `sbm threaded run` and `sbm batch run` commands along with the `--platinumdbsource="<database name>"` and `--platinumserversource="<server name>"` the app will generate a DACPAC from the source database then can then be used to run a build directly on your target(s).
+1. From a DACPAC file using the `sbm scriptextract` command. This method leverages a DACPAC that was created against your "Platinum Database" (why platinum? because it's even more precious than gold!). The Platinum database should have the schema that you want all of your other databases to look like. (don't have a DACPAC created, don't worry, you can create one with the `sbm dacpac` command) Learn more about DACPACs and [data-tier applications](https://docs.microsoft.com/en-us/sql/relational-databases/data-tier-applications/data-tier-applications?view=sql-server-2017)  method.
+
+2. From a set of script files using `sbm create`. This command allows you to specify an `--outputsbm` file to be created and a list of `--scripts` files to populate the SBM file with.
+
+3. From an SBX file. What is this? An SBX file is an XML file in the format of the `SqlSyncBuildProject.xml` file (see above) that has an `.sbx` extension. When you use the `sbm package` command, it will read the `.sbx` file and create the `.sbm` file with the referenced scripts.
+
+4. An SBM package file can be created indirectly as well, using the `sbm threaded run` and `sbm batch run` commands along with the `--platinumdbsource="<database name>"` and `--platinumserversource="<server name>"` the app will generate a DACPAC from the source database then will then be used to generate an SBM at run time to build directly on your target(s).
+
+----
+
+## Targeting multiple databases
+
+You define your database update targets leveraging an `--override` file or leverage Azure Service Bus Topics when using Azure Batch. The details of database targeting can be found [here](docs/override_options.md#database-targeting-option)
 
 ----
 
@@ -99,7 +113,7 @@ Using the `sbm threaded run` command will allow for updating multiple databases
 
 ### **Batch**
 
-Using the `sbm batch query` command leverages Azure Batch to permit massively parallel updates across thousands of databases. To leverage Azure Batch, you will first need to set up your Batch account. The instructions for this can be found [here](docs/azure_batch.md). 
+Using the `sbm batch query` command leverages Azure Batch to permit massively parallel updates across thousands of databases. To leverage Azure Batch, you will first need to set up your Batch account. The instructions for this can be found [here](docs/azure_batch.md).
 An excellent tool for viewing and monitoring your Azure batch accounts and jobs can be found here [https://azure.github.io/BatchExplorer/](https://azure.github.io/BatchExplorer/)
 
 ----
@@ -108,11 +122,10 @@ An excellent tool for viewing and monitoring your Azure batch accounts and jobs
 
 In addition to using Sql Build Manager to perform database updates, you can also run SELECT queries across all of your databases to collect data. In the case of both `threaded` and `batch` a consolidated results file is saved to the location of your choice
 
-### **Threaded**
+### Threaded
 
 Using the `sbm threaded query` command will allow for querying multiple databases in parallel, but still executed from the local machine.
 
-### **Batch**
+### Batch
 
 Using the `sbm batch query` command leverages Azure Batch to permit massively parallel queries across thousands of databases. (For information on how to get started with Azure Batch, go [here](docs/azure_batch.md))
-
diff --git a/docs/change_notes.md b/docs/change_notes.md
@@ -2,12 +2,20 @@
 # SQL Build Manager Change Notes
 
 
+### Version 13.0.3
+
+- *FIXED:* Update to Dacpac change scripts to identify new header delimiter
+- *FIXED:* Issue creating scripts between incompatable SQL Server versions. Will now output a warning and continue to create the scripts with the flag `AllowIncompatiblePlatform=true` 
+- *ADDED:* New command `sbm create` to create a new SBM file from a list of scripts
+- *UPDATED:* Can now use Windows auth for DACPAC creation
+- *UPDATED:* updated Nuget packages
+
 ### Version 13.0.2
 
 - *FIXED:* Update to ensure all Queue messages are retrieved efficiently
 - *ADDED:* New utility method `sbm batch dequeue` to remove all messages from the Service Bus Queue topic (without processing them)
 - *UPDATED:* Code clean up and refactoring to accommodate latest version of System.CommandLine
-- *FIXED:* Issue with SQL text syntax higlighting formatting in .NET 5.0
+- *FIXED:* Issue with SQL text syntax highlighting formatting in .NET 5.0
 - **NOTE:** Removed "Construct Command Line" menu options from Windows UI. Users should leverage the generated help docs for sbm.exe
 
 ### Version 13.0.1
@@ -23,7 +31,7 @@
 
 ### Version 12.1.0
 
-- *UPDATED:* Removed log4net logging. Unified logging via ILogger created in SqlBuildManager.Loggging. Implements Serilog
+- *UPDATED:* Removed log4net logging. Unified logging via ILogger created in SqlBuildManager.Logging. Implements Serilog
 
 ### Version 12.0.0
 

diff --git a/docs/local_build.md b/docs/local_build.md
@@ -1,6 +1,6 @@
 # Local Builds
 
-If you have a single databases to update, you can use a local build. 
+If you have a single databases to update, you can use a local build.
 
 Sample command:
 

diff --git a/docs/override_options.md b/docs/override_options.md
@@ -25,7 +25,7 @@ Why `client`?  Inside the `.sbm` file, there is a default database target set to
 
  This is why the flag name is `--override`!
 
- ### Runtime
+### Runtime
 
 If the `--override` setting is provided but there isn't a `--servicebustopicconnection` value, the runtime will use this file directly to update the database and/or distribute traffic for `threaded` and `batch`. It is important to also understand the impact of [concurrency options](concurrency_options.md).
 

diff --git a/docs/threaded_build.md b/docs/threaded_build.md
@@ -12,5 +12,4 @@ sbm threaded run
     --username "dbusername" ^
     --password "dbpassword" ^
     --override "targetdatabases.cfg" ^
-
-```
+```
diff --git a/src/AssemblyVersioning.cs b/src/AssemblyVersioning.cs
@@ -25,7 +25,7 @@
 //   These can be found in SqlBuildManager.Setup -> Organize Your Setup -> General Information
 // ** Also, don't forget to update the change_notes.xml and .html files!
 
-[assembly: AssemblyVersion("13.0.2")]
-[assembly: AssemblyFileVersion("13.0.2")]
+[assembly: AssemblyVersion("13.0.3")]
+[assembly: AssemblyFileVersion("13.0.3")]
 
 
diff --git a/...BuildManager.Console.Dependent.UnitTest/SqlBuildManager.Console.Dependent.UnitTest.csproj b/...BuildManager.Console.Dependent.UnitTest/SqlBuildManager.Console.Dependent.UnitTest.csproj
@@ -6,7 +6,7 @@
     <Deterministic>false</Deterministic>
   </PropertyGroup>
   <PropertyGroup>
-    <NoWarn>1701;1702</NoWarn>
+    <NoWarn>1701;1702;NU1608</NoWarn>
   </PropertyGroup>
   <ItemGroup>
     <Compile Include="..\AssemblyVersioning.cs" Link="Properties\AssemblyVersioning.cs" />
@@ -47,9 +47,9 @@
 
     <PackageReference Include="Microsoft.Extensions.Logging" Version="5.0.0" />
 
-    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.4" />
-    <PackageReference Include="MSTest.TestAdapter" Version="2.2.3" />
-    <PackageReference Include="MSTest.TestFramework" Version="2.2.3" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
+    <PackageReference Include="MSTest.TestAdapter" Version="2.2.5-preview-20210605-01" />
+    <PackageReference Include="MSTest.TestFramework" Version="2.2.5-preview-20210605-01" />
     <PackageReference Include="coverlet.collector" Version="3.0.3">
       <PrivateAssets>all</PrivateAssets>
       <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>

diff --git a/src/SqlBuildManager.Console.ExternalTest/CliTests.cs b/src/SqlBuildManager.Console.ExternalTest/CliTests.cs
@@ -25,7 +25,6 @@ public class CliTests
         private string linuxSettingsFilePath;
         private string settingsFileKeyPath;
 
-        private string logFile;
         [TestInitialize]
         public void ConfigureProcessInfo()
         {

diff --git a/src/SqlBuildManager.Console.ExternalTest/SqlBuildManager.Console.ExternalTest.csproj b/src/SqlBuildManager.Console.ExternalTest/SqlBuildManager.Console.ExternalTest.csproj
@@ -3,11 +3,14 @@
     <TargetFrameworks>netcoreapp3.1;net5.0</TargetFrameworks>
     <IsPackable>false</IsPackable>
   </PropertyGroup>
+  <PropertyGroup>
+    <NoWarn>1701;1702;NU1608</NoWarn>
+  </PropertyGroup>
   <ItemGroup>
     <PackageReference Include="Microsoft.Extensions.Logging" Version="5.0.0" />
-    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.4" />
-    <PackageReference Include="MSTest.TestAdapter" Version="2.2.3" />
-    <PackageReference Include="MSTest.TestFramework" Version="2.2.3" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
+    <PackageReference Include="MSTest.TestAdapter" Version="2.2.5-preview-20210605-01" />
+    <PackageReference Include="MSTest.TestFramework" Version="2.2.5-preview-20210605-01" />
     <PackageReference Include="coverlet.collector" Version="3.0.3">
       <PrivateAssets>all</PrivateAssets>
       <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
@@ -29,9 +32,17 @@
     <Content Remove="..\TestConfig\sbm-linux-11.4.0.zip" />
     <Content Remove="..\TestConfig\sbm-linux-12.1.0.zip" />
     <Content Remove="..\TestConfig\sbm-linux-12.2.0.zip" />
+    <Content Remove="..\TestConfig\sbm-linux-13.0.0.zip" />
+    <Content Remove="..\TestConfig\sbm-linux-13.0.1.zip" />
+    <Content Remove="..\TestConfig\sbm-linux-13.0.2.zip" />
+    <Content Remove="..\TestConfig\sbm-linux-13.1.0.zip" />
     <Content Remove="..\TestConfig\sbm-windows-11.4.0.zip" />
     <Content Remove="..\TestConfig\sbm-windows-12.1.0.zip" />
     <Content Remove="..\TestConfig\sbm-windows-12.2.0.zip" />
+    <Content Remove="..\TestConfig\sbm-windows-13.0.0.zip" />
+    <Content Remove="..\TestConfig\sbm-windows-13.0.1.zip" />
+    <Content Remove="..\TestConfig\sbm-windows-13.0.2.zip" />
+    <Content Remove="..\TestConfig\sbm-windows-13.1.0.zip" />
   </ItemGroup>
   <ItemGroup>
     <Compile Update="Properties\Resources.Designer.cs">

diff --git a/src/SqlBuildManager.Console.UnitTest/SqlBuildManager.Console.UnitTest.csproj b/src/SqlBuildManager.Console.UnitTest/SqlBuildManager.Console.UnitTest.csproj
@@ -5,6 +5,9 @@
     <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
     <Deterministic>false</Deterministic>
   </PropertyGroup>
+  <PropertyGroup>
+    <NoWarn>1701;1702;NU1608</NoWarn>
+  </PropertyGroup>
   <ItemGroup>
     <Compile Include="..\AssemblyVersioning.cs" Link="Properties\AssemblyVersioning.cs" />
   </ItemGroup>
@@ -20,9 +23,9 @@
   </ItemGroup>
   <ItemGroup>
     <PackageReference Include="Microsoft.Extensions.Logging" Version="5.0.0" />
-    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.4" />
-    <PackageReference Include="MSTest.TestAdapter" Version="2.2.3" />
-    <PackageReference Include="MSTest.TestFramework" Version="2.2.3" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
+    <PackageReference Include="MSTest.TestAdapter" Version="2.2.5-preview-20210605-01" />
+    <PackageReference Include="MSTest.TestFramework" Version="2.2.5-preview-20210605-01" />
     <PackageReference Include="coverlet.collector" Version="3.0.3">
       <PrivateAssets>all</PrivateAssets>
       <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>

diff --git a/src/SqlBuildManager.Console/Batch/BatchCleaner.cs b/src/SqlBuildManager.Console/Batch/BatchCleaner.cs
@@ -21,7 +21,6 @@
 using System.Text;
 using System.Text.RegularExpressions;
 using System.Threading.Tasks;
-using System.Linq;
 using MoreLinq;
 namespace SqlBuildManager.Console.Batch
 {

diff --git a/src/SqlBuildManager.Console/Batch/BatchExecution.cs b/src/SqlBuildManager.Console/Batch/BatchExecution.cs
@@ -94,7 +94,7 @@ public Execution(CommandLineArgs cmdLine, string queryFile, string outputFile) :
                 log.LogInformation($"Extracting Platinum Dacpac from {cmdLine.DacPacArgs.PlatinumServerSource} : {cmdLine.DacPacArgs.PlatinumDbSource}");
                 string dacpacName = Path.Combine(cmdLine.RootLoggingPath, cmdLine.DacPacArgs.PlatinumDbSource + ".dacpac");
 
-                if (!DacPacHelper.ExtractDacPac(cmdLine.DacPacArgs.PlatinumDbSource, cmdLine.DacPacArgs.PlatinumServerSource, cmdLine.AuthenticationArgs.UserName, cmdLine.AuthenticationArgs.Password, dacpacName))
+                if (!DacPacHelper.ExtractDacPac(cmdLine.DacPacArgs.PlatinumDbSource, cmdLine.DacPacArgs.PlatinumServerSource, cmdLine.AuthenticationArgs.AuthenticationType, cmdLine.AuthenticationArgs.UserName, cmdLine.AuthenticationArgs.Password, dacpacName))
                 {
                     log.LogError($"Error creating the Platinum dacpac from {cmdLine.DacPacArgs.PlatinumServerSource} : {cmdLine.DacPacArgs.PlatinumDbSource}");
                 }

diff --git a/src/SqlBuildManager.Console/CommandLine/CommandLineArgs.cs b/src/SqlBuildManager.Console/CommandLine/CommandLineArgs.cs
@@ -110,6 +110,8 @@ public string Override {
         [JsonIgnore]
         public string OutputSbm { get; set; }
         [JsonIgnore]
+        public FileInfo[] Scripts { get; set; }
+        [JsonIgnore]
         public string DacpacName
         {
             get