Moving some subsets of commands in dbatools 2.0

[potatoqualitee]

I've often been asked to separate dbatools into different modules, similar to other large modules.

I've resisted because it's hard to categorize commands and there's so much interrelation.

But now I see there are a couple that are very easy to separate. Those include:

• The Performance Monitor commands

  • Add-DbaPfDataCollectorCounter
  • Export-DbaPfDataCollectorSetTemplate
  • Get-DbaPfAvailableCounter
  • Get-DbaPfDataCollector
  • Get-DbaPfDataCollectorCounter
  • Get-DbaPfDataCollectorCounterSample
  • Get-DbaPfDataCollectorSet
  • Get-DbaPfDataCollectorSetTemplate
  • Import-DbaPfDataCollectorSetTemplate
  • Invoke-DbaPfRelog
  • Remove-DbaPfDataCollectorCounter
  • Remove-DbaPfDataCollectorSet
  • Start-DbaPfDataCollectorSet
  • Stop-DbaPfDataCollectorSet

• The XESmartTarget commands

  • Get-DbaXESmartTarget
  • New-DbaXESmartCsvWriter
  • New-DbaXESmartEmail
  • New-DbaXESmartQueryExec
  • New-DbaXESmartReplay
  • New-DbaXESmartTableWriter
  • Remove-DbaXESmartTarget
  • Start-DbaXESmartTarget
  • Stop-DbaXESmartTarget

• The Data Masking and Data Generation commands

  • Invoke-DbaDbDataMasking
  • Invoke-DbaDbPiiScan
  • New-DbaDbMaskingConfig
  • Test-DbaDbDataMaskingConfig
  • Get-DbaRandomizedDataset
  • Get-DbaRandomizedDatasetTemplate
  • Get-DbaRandomizedType
  • Get-DbaRandomizedValue
  • Invoke-DbaDbDataGenerator
  • New-DbaDbDataGeneratorConfig
  • Test-DbaDbDataGeneratorConfig

• The Windows Server Failover Cluster commands

  • Get-DbaWsfcAvailableDisk
  • Get-DbaWsfcCluster
  • Get-DbaWsfcDisk
  • Get-DbaWsfcNetwork
  • Get-DbaWsfcNetworkInterface
  • Get-DbaWsfcNode
  • Get-DbaWsfcResource
  • Get-DbaWsfcResourceType
  • Get-DbaWsfcRole
  • Get-DbaWsfcSharedVolume

• The Replication commands

  • Export-DbaRepServerSetting
  • Get-DbaRepDistributor
  • Get-DbaRepPublication
  • Get-DbaRepServer
  • Test-DbaRepLatency

Maybe we divide it up by the categories on dbatools.io/commands but combine many of them as .core and .utility? How far do we go? Just the above or the whole way?

This will help dbatools import a lot faster and if you install dbatools, it'll come with all of these, but when you execute Get-DbaDatabase, it'll only load dbatools-core behind the scenes. If you run Get-DbaRepDistributor, it'll load dbatools.core and dbatools.replication submodule. If you want to just install dbatools.replication, it'll be dbatools.replication and dbatools.core.

@niphlod would your script that gets interrelated commands be able to tell us what needs to go in dbatools.core? Basically, what commands will be commonly used in the above groups.

Note: I'll soon be submitting a giant PR with my library changes and will love if anyone has the time to review.

/cc @niphlod @sanderstad @andreasjordan @jpomfret @wsmelton @SQLDBAWithABeard @ClaudioESSilva @lancasteradam @nvarscar

(anyone is welcome to contribute -- I just pinged people who are around, whereas others have contributed heavily but aren't around as much.)

[nvarscar]

This has been my dream for long, I'd happily contribute given I can find time. I think best results, from the module size perspective, can be achieved by separating not just by command type, but also by the underlying libraries (dlls) that support said commands. As an example, replication and DacPac commands both rely on certain dlls that are not invoked anywhere else. The general rule of dbatools was "load everything right away", but with this approach we can shift towards on-demand dll load, which would definitely improve module load time. It can either be split into independent modules, or, at the very least, hide dynamic load in an internal function that calls [System.AppDomain]::CurrentDomain.GetAssemblies().GetName() to check for loaded classes. Example

If we had a script that somehow discovered all dependencies for each command, we could then easily group them by requirements.

[potatoqualitee]

Great feedback! DacFX is a great candidate to separate out. I'll ping you on the PR to see our new library (which I'll rename to dbatools.library). Right now, the primary libraries are SMO, SqlClient, Replication and DacFx. The XE commands seem now firmly in SMO.

Would you be happy with these in dbatools.deployment?

• Get-DbaInstalledPatch
• Get-DbaKbUpdate
• Install-DbaInstance
• Invoke-DbaAdvancedInstall
• Invoke-DbaAdvancedUpdate
• New-DbaDacOption
• New-DbaDacProfile
• Publish-DbaDacPackage
• Save-DbaKbUpdate
• Update-DbaInstance

[potatoqualitee]

Oh, oof. Okay so now repositories. The only one that maybe could separate out is the library repo which currently looks like this: https://github.com/dataplat/dbatools-library/tree/ci

[SQLDBAWithABeard]

Option 1

Keep the dbatools repository as is

Move Build and Deploy of individual modules to their own directory

• Will make overall testing easier,
• Will enable smaller more specific modules.
• Testing can be Module specific as well as dependency specific
• Reduce complexity in development as everything is in one place
• Single place for issues and discussions

Option 2

Split dbatools repository into separate repos for types/themes/categories (naming ?)

• Will make testing for specific themes easier
• Will enable smaller more specific modules.
• Testing can be Module specific
• Dependency specific testing will require more upfront effort ( testing changes to dbatools-library for example to ensure they dont break dbatools-hadr)
• Development effort a little harder will require documentation and understanding of the landscape (some people will need words, some will need images, some will do better with video)
• Confusion with issues/discussions and short links to issues and discussions. (possibly easiest is to keep dbatools issues and use GitHub Actions to create/update to and from the relevant repo ??

I am sure there is more but that was a quick off the top of my head.

But I think it is a great idea and can we also deploy only the required files to PowerShell Gallery when we make this change? So we dont have to download all the function ps1 files or the devcontainer files for example ;-)

[potatoqualitee]

You make a great point about the issues being in one place. I was leaning heavy to simplify development for others and that pretty much convinced me, thank you 💯 I know we can disable issues in other repos, but that's just showing how complex things can be.

Oh yeah! the dev container too. For sure, this is going to be all-in-one.

[potatoqualitee]

Except for the library 😅 That one will stay in its own repo

[wsmelton]

Yes, please keep it in one repo. Dealing with separate repos can be a pill both for the user and CI/CD.

If we are going to publish them all individually to the PS Gallery it helps the build process be easier to maintain.

[jpomfret]

Makes sense to me to split them out into separate modules - as for working out all the dependencies that is going to be hard since there is so many that interconnect.

Thinking out loud but not fully at this point - do we currently have the right pester tests for each command to ensure everything works after the spilt

• calling the function should tell us if it's missing anything
• do we also need to run tests when only loading the sub modules.. i.e. load just dbatools.replication and then make sure all the commands still work?

[potatoqualitee]

Great question, @jpomfret ! @niphlod, I know @andreasjordan uses your "do" a lot. Can we do for submodules and then run the full suite on "dbatools.core" or whatever it'll be named? Sucks that core means two things -- .net core and the core necessity.

I'm mildly considering the base module be named dbatools.base instead of dbatools.core, what do yall think? The two libraries will be named datools.library and dbatools.core.library

[niphlod]

the logic for "which tests should we run" can definitely make decision based on some parameters, we just need to alter it according to the new structure. Right now is basically a "if there is functions/Get-Something.ps1 let's run tests/Get-Something.ps1" , The "do" just does a "-like" and restrict further the set of tests to be launched

[wsmelton]

I would say this will be opportune time to rewrite tests to Pester 5.

There are many that depends on the current structure of the repo so when it changes they have to be touched anyway.

[jpomfret]

Would agree on the pester 5... but that is another big chunk of work to make sure we get right.

[potatoqualitee]

yeah, someone else can do Pester 5 and it'd all have to be done within the month 😅 (I never plan to do it unless AI writes me a rewriter)

[wsmelton]

Can comment briefly at this point (on mobile)...

Your build and test process have to consider whether things play nice together with SMO. If a user imports one sub module over the whole module initially or in the same session.

The challenge with SMO is identifying the core libraries it has not necessarily our core commands. Certain DLLs cause a core set of SMO namespaces to be imported or required. We have to ensure that plays nice with how the sub modules would both function together but also individually.

I would recommend identifying and mapping out the dependencies that SMO has with it's libraries. That is going to affect the import process and tests ability. We need to make sure we don't try to import things that are not already present or forget to package a library with a sub module.

[potatoqualitee]

So far, I think I'm only going to divide up the commands by whole nuget package. For instance, dbatools.core will import all SMO and SqlClient files. Currently, this takes about 179ms, I believe, so it's not a problem. The Dac commands will import the DacFx nuget package, etc. I wrote an assembly resolver that will take care of version mismatches. You can check it out here:

dbatools/bin/library.ps1

Lines 20 to 58 in d3b7fde

	protected Assembly AssemblyResolve(object sender, ResolveEventArgs e)
	{
	string[] dlls = {
	"System.Runtime.CompilerServices.Unsafe",
	"System.Resources.Extensions",
	"Microsoft.SqlServer.ConnectionInfo",
	"Microsoft.SqlServer.Smo",
	"Microsoft.Identity.Client",
	"System.Diagnostics.DiagnosticSource",
	"Microsoft.IdentityModel.Abstractions",
	"Microsoft.Data.SqlClient",
	"System.Configuration.ConfigurationManager",
	"Microsoft.SqlServer.Replication",
	"Microsoft.SqlServer.Rmo",
	"System.Private.CoreLib"
	};
	var name = new AssemblyName(e.Name);
	var assemblyName = name.Name.ToString();
	foreach (string dll in dlls)
	{
	if (assemblyName == dll)
	{
	string filelocation = "$dir" + dll + ".dll";
	//Console.WriteLine(filelocation);
	return Assembly.LoadFrom(filelocation);
	}
	}
	foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
	{
	// maybe this needs to change?
	var info = assembly.GetName();
	if (info.FullName == e.Name) {
	return assembly;
	}
	}
	return null;
	}