-
-
Notifications
You must be signed in to change notification settings - Fork 473
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LastExitCode modified by an executable that is run as part of a test and then returned by Pester resulting in CI build failure #1616
Comments
Not sure if Pester's responsibility to mask $LastExitCode of another process. Warning might make sense though or something added to detailed diagnostics, I've been caught with this before too and it's hard to find what's causing it. Is $LastExitCode 1 considered an exception which Should -Throw? Probably not? Should ErrorLevel/Exit codes be masked by Pester tests? As a test framework, I would expect artifacts be torn down and state reset to original post test. There's some TODOs around this Had a similar issue with git.exe throwing errors for things that weren't errors in our build pipelines. I see that this mock example would set the last exit code, perhaps it could be reset in your test script after calling choco? function choco.exe {}
Mock -CommandName choco.exe -MockWith { $Global:LASTEXITCODE = 1 } #415 has a workaround. function Invoke-ExternalCommand {
# Reset $result in case it's used somewhere else
$result = $null
# Reset $LASTEXITCODE in case it was tripped somewhere
$Global:LASTEXITCODE = 0
$result = & $command $arguments
if ($LASTEXITCODE -ne 0) {
Throw "Something bad happened while executing $command. Details: $($result | Out-String)"
}
Write-Verbose "$($result | Out-String)"
}
# Called from other functions like this:
Invoke-ExternalCommand MSTest.exe @(
"/testcontainer:$testContainer",
"/resultsfile:$outputFile") [x] usePackageExitCodes - Use Package Exit Codes - Package scripts can provide exit codes. With this on, package exit codes will be what choco uses for exit when non-zero (this value can come from a dependency package). Chocolatey defines valid exit codes as 0, 1605, 1614, 1641, 3010. With this feature off, choco will exit with 0, 1, or -1 (matching previous behavior). Available in 0.9.10+. |
Hmm yeah, Pester should exit with 0 as long as no test is failed, so we should probably set the exit code to 0 explicitly. As for asserting LastExitCode != 0, that seems like something that could be bolted on -Throw, but I can't think of a good name for the switch now. @asears You posted link to a TODO, but it links to the whole file which has few thousands of lines, which one did you mean? |
Yeah, there's a few TODOs in that file. Don't know if any affect this particular issue.
https://github.com/pester/Pester/blob/v5.0/src/Pester.Runtime.psm1#L247
https://github.com/pester/Pester/blob/v5.0/src/Pester.Runtime.psm1#L1966
https://github.com/pester/Pester/blob/v5.0/src/Pester.Runtime.psm1#L1689
https://github.com/pester/Pester/blob/v5.0/src/Pester.Runtime.psm1#L1696 The last one was the one that caught my eye as something to check. Looking at them now, I don't know that any of these are particularly relevant for this issue. As a side note, might be worthwhile removing some of the commented code lines from codebase and putting those in another branch, or using a feature toggle to ease testing & debugging. Would this particular "feature" be clearing any executable error codes for the test run and returning a status code / errorlevel of 0 if all tests are successful? -ResetLastExitCodeVariable ? Is it Pester's business to reset the LastExitCode Environment Variable at all or something to just warn on, or should -throw in a strict test if $LASTEXITCODE != 0? Currently it seems to be only used in the build and release scripts internal to Pester. Might be worthwhile to look at how this should behave in Azure Devops pipeline task or see if it could just be warned on after an executable is run, and set after Invoke-Pester is run vs. changing functionality. I think the problem is not that it isn't being captured or reset on exit, it's that it's hard to discover what exe is causing this issue and the test isn't failing due to LASTEXITCODE being 1, but the Devops Pipeline is. It's a case of "polluted environment?" There's some issues around this in choco. https://github.com/chocolatey/choco/search?q=LastExitCode&type=Issues The workaround for choco is choco -v. There might be a fix in progress though that was mentioned last year. It would be good to test that all environment variables and state before and after a Pester run are equal, and Pester-specific internal variables are disposed of after each test in an Azure Devops pipeline and new variables are surfaced somehow. There were some warnings from PSScriptAnalyzer about reusing reserved variables. Was seeing some issues with PesterConfiguration but nothing I could repro. |
I thought ExitCodes wasn't really a PowerShell thing, but more of a backwards compatibility feature for legacy-applications. Maybe I'm wrong, but keeping that assumption in mind...
Not sure if you meant to always exit or only when using Lines 1135 to 1137 in 8d55526
Won't a second assertion in the test cover this scenario? If they care to check it, they might as well do it explicitly.
|
@fflaten I mean setting the exit code explicitly ( As for the assertion, yeah that is possible with |
1. General summary of the issue
I have been writing some infrastructure tests and to test that chocolatey was installed I was doing this:
My tests were all passing, but when running in my Packer build were causing the build to fail because the Exit Code was non-zero. I eventually tracked it down to the above test and have discovered that choco sets the
$LastExitCode
variable to 1 even though it has effectively executed successfully (note it only seems to do this when usingchoco
on its own, rather than with a specific command).It seems that because none of the tests had failed, Pester wasn't setting $LastExitCode at all and so it was just being passed through with whatever choco had set it to.
I've worked around this by testing for the install of choco by just looking for the presence of the .exe on disk, but I wondered if this non-handling of
$LastExitCode
when all tests have passed was expected behaviour or perhaps something that should be addressed, because it caused me some confusion.2. Describe Your Environment
Pester version : 5.0.2 C:\Program Files\WindowsPowerShell\Modules\Pester\5.0.2\Pester.psd1
PowerShell version : 5.1.17763.1007
OS version : Microsoft Windows NT 10.0.17763.0
3. Expected Behavior
I expected
$LastExitCode
to always be 0 if all tests have passed.4.Current Behavior
$LastExitCode
is set to the exit code of an executable that is run as part of the tests.5. Possible Solution
Have Pester set
$LastExitCode
even if all tests have passed. I am assuming currently it only sets its value if failed.However there might be reasons not to do this and i'm not sure how much of a breaking change it might be for some use cases. The problem here is more choco than Pester for returning a non-zero exit code for a valid operation. I guess in most cases when an exe returns a non-zero exit code as part of a test run its perfectly valid for that to result in a failure state outside of the environment.
If the consensus is that this shouldn't really be solved, please close this issue but i'm going to raise it anyway as it might help someone who has similar in the future.
6. Context
Pester being used for infrastructure validation tests as part of a Packer VM image build in Azure.
The text was updated successfully, but these errors were encountered: