WaitADDomain: fails if DC not ready

[ghost]

WaitADDomain: fails if DC not ready

Verbose logs showing the problem

[EC2AMAZ-0J01R66]: LCM:  [ Start  Resource ]  [[WaitForADDomain]ForestWait]
[EC2AMAZ-0J01R66]: LCM:  [ Start  Test     ]  [[WaitForADDomain]ForestWait]
[EC2AMAZ-0J01R66]:                            [[WaitForADDomain]ForestWait] Determining the current state of the 
Active Directory domain 'tf-tf.local'. (WFADD0013)
[EC2AMAZ-0J01R66]:                            [[WaitForADDomain]ForestWait] Searching for a domain controller in the 
domain 'tf-tf.local'. (WFADD0001)
[EC2AMAZ-0J01R66]:                            [[WaitForADDomain]ForestWait] Impersonating the credentials 
'tf-tf\xxxxxxxxx' when looking for a domain controller. (WFADD0011)
[EC2AMAZ-0J01R66]:                            [[WaitForADDomain]ForestWait] Searching for a domain controller in the 
domain 'tf-tf.local'. (ADCOMMON0052)
[EC2AMAZ-0J01R66]:                            [[WaitForADDomain]ForestWait] The type 
'System.DirectoryServices.ActiveDirectory.DirectoryContext' is already loaded into the PowerShell session. 
(ADCOMMON0043)
[EC2AMAZ-0J01R66]:                            [[WaitForADDomain]ForestWait] Get a new Active Directory context of the 
type 'Domain'. (ADCOMMON0046)
[EC2AMAZ-0J01R66]:                            [[WaitForADDomain]ForestWait] The Active Directory context will target 
'tf-tf.local'. (ADCOMMON0047)
[EC2AMAZ-0J01R66]:                            [[WaitForADDomain]ForestWait] The Active Directory context will be 
accessed using the 'tf-tf\xxxxxxxxx' credentials. (ADCOMMON0048)
[EC2AMAZ-0J01R66]: LCM:  [ End    Test     ]  [[WaitForADDomain]ForestWait]  in 4.8180 seconds.
PowerShell DSC resource MSFT_WaitForADDomain  failed to execute Test-TargetResource functionality with error message: 
Exception calling "FindOne" with "1" argument(s): "Domain controller not found in the domain "tf-tf.local"." 
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : ProviderOperationExecutionFailure
    + PSComputerName        : localhost
 

Suggested solution to the issue

Wait until the DC is ready

The DSC configuration that is used to reproduce the issue (as detailed as possible)

I create new server and wait for DC to become ready to join the domain. If the DC is ready, server joins the domain without problem, but if WaitForADDomain runs when DC is being configured, the resource fails. During the next consistency check the server joins the domain as the DC is ready by that time (also proving the configuration is hopefully correct).

        WaitForADDomain ForestWait
        {
            DomainName = $InternalDomain
            WaitTimeout = 900
            Credential = $default_creds
            DependsOn = "[DNSServerAddress]DNS"
            WaitForValidCredentials = $true
        }

The operating system the target node is running

OsName               : Microsoft Windows Server 2019 Datacenter
OsOperatingSystemSKU : DatacenterServerEdition
OsArchitecture       : 64-bit
WindowsVersion       : 1809
WindowsBuildLabEx    : 17763.1.amd64fre.rs5_release.180914-1434
OsLanguage           : en-US
OsMuiLanguages       : {en-US}

Version and build of PowerShell the target node is running

Name                           Value
----                           -----
PSVersion                      5.1.17763.771
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17763.771
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version of the DSC module that was used ('dev' if using current dev branch)

4.2.0.0

[X-Guardian]

Hi @Crusad, looking at the Find-DomainController function, it looks like the FindOne() method call in Find-DomainControllerFindOneWrapper is returning an unexpected exception. To see what the exception is, can you add the following line to Modules/ActiveDirectoryDsc.Common/ActiveDirectoryDsc.Common.psm1 before line 2260 (throw $_) to output the exception name to the logs and then run the Dsc again.

Write-Verbose -Message "Unknown Exception: $($_.Exception.GetType().FullName)" -Verbose

[johlju]

Yes I agree, it looks like another error makes the resource not wait the 900 seconds (the timeout value) as it should 🤔

[X-Guardian]

Yes, I see @Crusad is using Server 2019, so I'm suspicious that the exception name may have changed that the function is catching on.

[ghost]

I will definitely test what @X-Guardian suggested, just testing this is a bit time-consuming for me and I was too busy past two days to get into it. I will get back to you with results. And yep, I am using 2019 (1809).

[X-Guardian]

No problem @Crusad.

@johlju, I was thinking that we should output the exception full name whenever we hit an unknown exception in any of the resources. It would make diagnosing these issues easier.

[xpn]

@X-Guardian So I've come across this issue myself when I'm attempting to create a new lab environment. This involves a server waiting for a domain controller to come online and results in the following entry being added to the "Desired State Configuration - Operational" event log:

Job {A84EF340-4741-11EA-8249-06C12448A1E2} : 
Message Exception calling "FindOne" with "1" argument(s): "Domain controller not found in the domain "lab.local"." 
HResult -2146233087 
StackTrack    at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
   at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
   at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync)
   at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke(IEnumerable input, PSInvocationSettings settings)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.ResourceProviderAdapter.ExecuteCommand(PowerShell powerShell, ResourceModuleInfo resInfo, String operationCmd, List`1 acceptedProperties, CimInstance nonResourcePropeties, CimInstance resourceConfiguration, LCMDebugMode debugMode, PSInvocationSettings pSInvocationSettings, UInt32& resultStatusHandle, Collection`1& result, ErrorRecord& errorRecord, PSModuleInfo localRunSpaceModuleInfo)

Reviewing the Test-TargetResource method, I have managed to recreate the issue with the following Powershell snippet:

$typeName = 'System.DirectoryServices.ActiveDirectory.DirectoryContext'

Add-Type -AssemblyName 'System.DirectoryServices'

$argumentList = @(
         "Domain"
         "doesnotexistyet.local"
     )

$objectParameters = @{
         TypeName     = $typeName
         ArgumentList = $argumentList
     }

$findDomain = New-Object @objectParameters

[System.DirectoryServices.ActiveDirectory.DomainController]::FindOne($findDomain)

I've also made the update to the exception handler as requested above and the following information is returned:

Unknown Exception: System.Management.Automation.MethodInvocationException

This is re-thrown from the following line where it is never handled from Test-TargetResource it seems:

ActiveDirectoryDsc/source/Modules/ActiveDirectoryDsc.Common/ActiveDirectoryDsc.Common.psm1

Line 2073 in 5ea5a13

throw $_

Please let me know if you require any further information to help recreate this issue.

[X-Guardian]

Hi @xpn. Thanks for the extra info. With your code snippet I can reproduce the error and see the problem. The exception thrown is of type System.Management.Automation.MethodInvocationException with an inner exception of type System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException. The current code only checks for a base exception of System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException.

Should be a simple case of adding a check to the Find-DomainController function for the ActiveDirectoryObjectNotFoundException within the current code that processes the MethodInvocationException.

I'm happy to review a PR if you want to submit one?

[X-Guardian]

OK I've done some more testing on this, and I think we still have an issue with a different exception.

Expanding @xpn's code snippet with a try/catch block:

$typeName = 'System.DirectoryServices.ActiveDirectory.DirectoryContext'

Add-Type -AssemblyName 'System.DirectoryServices'

$argumentList = @(
    "Domain"
    "doesnotexistyet.local"
)

$objectParameters = @{
    TypeName     = $typeName
    ArgumentList = $argumentList
}

$findDomain = New-Object @objectParameters

try
{
    [System.DirectoryServices.ActiveDirectory.DomainController]::FindOne($findDomain)
}
catch [System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException]
{
    Write-Host 'Specific catch: ActiveDirectoryObjectNotFoundException'
    Write-Host 'Exception Name: ' + $_.Exception.GetType().FullName
}
catch [System.Management.Automation.MethodInvocationException]
{
    Write-Host 'Specific catch: MethodInvocationException'
    Write-Host 'Exception Name: ' + $_.Exception.GetType().FullName
    Write-Host 'Inner Exception Name: ' + $_.Exception.InnerException.GetType().FullName
} 
catch
{
    Write-Host 'Generic catch'
    Write-Host 'Exception Name: ' + $_.Exception.GetType().FullName
    Write-Host 'Inner Exception Name: ' + $_.Exception.InnerException.GetType().FullName
}

For me, both on a 2012 R2 server running PowerShell 4 and a 2019 server running PowerShell 5.1, the above code hits the ActiveDirectoryObjectNotFoundException specific catch, even though the original outer exception is of type MethodInvocationException. You can test this for yourself by running the above code with and without the two specific catches.

We must therefore be hitting a different exception.

@xpn, if you are able to recreate this consistently within a DSC run, can you add the following code just before both throw statements in the Find-DomainController function so that we can capture both the outer and inner exception names:

Write-Verbose -Message "Exception Name: $($_.Exception.GetType().FullName)" -Verbose
Write-Verbose -Message "Inner Exception Name: $($_.Exception.InnerException.GetType().FullName)" -Verbose

This will hopefully give us the information we need to fix this.

[xpn]

Hmm, yeah I can recreate this with the code example above, similar if you remove the catch for the ActiveDirectoryObjectNotFoundException from your code, you then hit the MethodInvocationExcetpion catch... It's Schrodinger exception :)

I'll spin up the lab and test with your requested code and get back to you.

[jcuzzi]

@X-Guardian I added verbose logging on both throw statements in the Find-DomainController function. Here is the output when using the resource:

VERBOSE: [DC2]: LCM:  [ Start  Resource ]  [[WaitForADDomain]WaitForPrimaryDC]
VERBOSE: [DC2]: LCM:  [ Start  Test     ]  [[WaitForADDomain]WaitForPrimaryDC]
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] Determining the current state of the Active Directory domain 'testdomain.com'. (WFADD0013)
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] Searching for a domain controller in the domain 'testdomain.com'. (WFADD0001)
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] Impersonating the credentials 'testdomain\Administrator' when looking for a domain controller. (WFADD0011)
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] Searching for a domain controller in the domain 'testdomain.com'. (ADCOMMON0052)
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] The type 'System.DirectoryServices.ActiveDirectory.DirectoryContext' is already loaded into the PowerShell session. (ADCOMMON0043)
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] Get a new Active Directory context of the type 'Domain'. (ADCOMMON0046)
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] The Active Directory context will target 'testdomain.com'. (ADCOMMON0047)
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] The Active Directory context will be accessed using the 'testdomain\Administrator' credentials. (ADCOMMON0048)
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] Exception Name: System.Management.Automation.MethodInvocationException
VERBOSE: [DC2]:                            [[WaitForADDomain]WaitForPrimaryDC] Inner Exception Name: System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException
VERBOSE: [DC2]: LCM:  [ End    Test     ]  [[WaitForADDomain]WaitForPrimaryDC]  in 12.5290 seconds.
PowerShell DSC resource MSFT_WaitForADDomain  failed to execute Test-TargetResource functionality with error message: Exception calling "FindOne" with "1" argument(s): "Domain controller not found in the domain "testdomain.com"."
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : ProviderOperationExecutionFailure
    + PSComputerName        : DC2

VERBOSE: [DC2]: LCM:  [ End    Set      ]
The SendConfigurationApply function did not succeed.
    + CategoryInfo          : NotSpecified: (root/Microsoft/...gurationManager:String) [], CimExce
   ption
    + FullyQualifiedErrorId : MI RESULT 1
    + PSComputerName        : DC2

VERBOSE: Operation 'Invoke CimMethod' complete.
VERBOSE: Time taken for configuration job to complete is 29.308 seconds

[X-Guardian]

Cheers @jcuzzi, you are right, having the FindOne method call in a function does affect the catch handling of any exception from it. Here is the above test code modified with the FindOne call in a function:

function Find-DomainControllerFindOneWrapper
{
    [CmdletBinding()]
    [OutputType([System.DirectoryServices.ActiveDirectory.DomainController])]
    param
    (
        [Parameter(Mandatory = $true)]
        [System.DirectoryServices.ActiveDirectory.DirectoryContext]
        $DirectoryContext
    )

    [System.DirectoryServices.ActiveDirectory.DomainController]::FindOne($DirectoryContext)
}

$typeName = 'System.DirectoryServices.ActiveDirectory.DirectoryContext'

Add-Type -AssemblyName 'System.DirectoryServices'

$argumentList = @(
    "Domain"
    "doesnotexistyet.local"
)

$objectParameters = @{
    TypeName     = $typeName
    ArgumentList = $argumentList
}

$findDomain = New-Object @objectParameters

try
{
    Find-DomainControllerFindOneWrapper -DirectoryContext $findDomain
}
catch [System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException]
{
    Write-Host 'Specific catch: ActiveDirectoryObjectNotFoundException'
    Write-Host 'Exception Name: ' + $_.Exception.GetType().FullName
}
catch [System.Management.Automation.MethodInvocationException]
{
    Write-Host 'Specific catch: MethodInvocationException'
    Write-Host 'Exception Name: ' + $_.Exception.GetType().FullName
    Write-Host 'Inner Exception Name: ' + $_.Exception.InnerException.GetType().FullName
} 
catch
{
    Write-Host 'Generic catch'
    Write-Host 'Exception Name: ' + $_.Exception.GetType().FullName
    Write-Host 'Inner Exception Name: ' + $_.Exception.InnerException.GetType().FullName
}

For me, the exception is now caught by the MethodInvocationException catch instead of the ActiveDirectoryObjectNotFoundException. I'm going to take a look at the two exceptions and see what the difference is.