Set-RetryAfterReboot (chocolatey-postinstaller-checks.extension) & chocolatey-preinstaller-checks.extension

[bcurran3]

Discussion started at chocolatey/choco#544 (comment)

Intent:
Create a helper that, when used, would check a program's exit code and in the case of common MSI abortions; i.e. 1603, would create a scheduled task to further attempt (once) an installation of that package upon reboot.

Method - needs to be tested, but most likely would work.
schtasks.exe /create /f /tn Retry%ChocolateyPackageName% /ru SYSTEM /sc ONSTART /tr "choco install %ChocolateyPackageName% && schtasks.exe /delete /f /tn Retry%ChocolateyPackageName%"

Thoughts:
Would need some sort of flag so that subsequent failures don't enter an infinite loop attempting reinstallation at reboot forever. Could be implemented by dumping a simple RETRY.YES in the package's lib folder (Rob will hate that idea.) or possibly create a chocolatey-fastanswers.extension registry key somewhere to be checked, i.e. packagename could be set as a subkey with a packagename=1 to attempt reinstallation and Set-RetryAfterReboot would delete it and NOT configure the scheduled task to try again under that condition that it's already been retried.

Usage:
Get-PendingReboot
Install-ChocolateyPackage @packageArgs
Set-RetryAfterReboot

Note:
This would be a package for the community repository and "normal" users, and would be noted that it may not be compatible with some business environments using other management programs.

[DarwinJS]

Retry loop can be handled by not using the "&&" operator which only deletes the job if the first command succeeds. Use "&" instead so that the delete operation is always performed. So it would be "retry after reboot one time". This also would handle cases common cases like the chocolatey package is located on a network share that the system account can't get to.

Sometimes machines aren't rebooted very often - so testing what happens if two or three of these are cued up at once would be necessary - or as an alternative only allow one of these to cue up at a time (by keeping the task name the same no matter what package is doing the cue up.

Just as a side point, for the common use case of MSI 1618 (another install is in progress) - it is better to do a retry loop right at the time that code is received - but double check the chocolatey code in case chocolatey already does this internally.

Be prepared for a lot of debug requests against why this does not work in various circumstances ;)

[bcurran3]

Retry loop can be handled by not using the "&&" operator which only deletes the job if the first command succeeds.

The loop isn't caused by this. The loop would be caused by if the package failed installation again for same or different reasons as the installing the package would check and run Set-RetryAfterReboot again in the package installation script ad nauseum. That's why some sort of external retry count flag would be needed for each failed package. (Most likely registry entry. Would also make it easy do things like set retries to 3 if desired.)

Use "&" instead so that the delete operation is always performed.

You're 100% correct - thanks for pointing that out!

...is located on a network share that the system account can't get to.

I'm running with the idea that this is for the community repo where all packages would be downloaded over the Internet from the community repo. Businesses with internalized or custom packages would have to solve this problem on their own - they should have the manpower! If the package fails attempted re-installation, it's only taking a few CPU cycles away from their unknown Bitcoin miner running in the background - JK, LOL, :). I'm guessing they might make it work with yet another scheduled task run before all this that would simply map a drive letter to a fileshare that the system account would then be able to read. I'm running with the assumption that this mapping would stay available to the subsequent scheduled tasks run by the system account. I've got a feeling you could most likely pass or fail this concept.

Alternatively the above could be managed by timed scheduled events versus them happening on reboot.

testing what happens if two or three of these are cued up at once would be necessary

I can foresee if multiple scheduled tasks to install packages run at the same time, choco could step all over itself. From my experience choco.exe's clean up routines are not very multiple instance friendly. Skip ahead to BETTER PROPOSED SOLUTION - ONE TASK AND INSTALLING FROM A PACKAGES.CONFIG if you're in a rush.

CURRENTLY DISCUSSING PROPOSED SOLUTION - MULTIPLE INDIVIDUAL TASKS:

schtasks.exe /create /f /tn Retry%ChocolateyPackageName% /ru SYSTEM /sc ONSTART /tr "choco install %ChocolateyPackageName% & schtasks.exe /delete /f /tn Retry%ChocolateyPackageName%" ...

Here's a realistic hypothetical situation: Let's say you manually attempt to install googlechrome and it fails installation (which it's been doing a lot recently) and 5 other packages out of a 100 fail as well. Assuming all these packages are using the currently non-existent-but-being-discussed Set-RetryAfterReboot helper in their chocolateyinstall.ps1 after the Install-ChocolateyPackage command.... six tasks would be created to attempt installation of each package individually at reboot . Maybe two weeks have passed and computer finally reboots. Six choco install packagename commands would occur. They would each run through the same (or updated) chocolateyinstall.ps1. They hit Set-RetryAfterReboot after Install-ChocolateyPackage. Set-RetryAfterReboot would check if it was a retry by checking the flag. If the flag existed it would not setup a subsequent retry. Meanwhile the retried packages would either install, not install, or announce to the Phantom Zone that package x.x. is already installed. I don't see harm or problems in any of these scenarios. Sure, it's possible the package doesn't get installed, but a decent attempt was made. It is likely though that choco.exe will "get confused" with multiple instances running.

BETTER PROPOSED SOLUTION - ONE TASK AND INSTALLING FROM A PACKAGES.CONFIG
I must have been too LASER focused last night... because I did think about this when making Chocolatey Fast Answers Extension based on my previous experiences making Choco Package List Backup to Local and Cloud and more so Choco Persistent Packages.

The better solution would be for the non-existent-but-being-discussed Set-RetryAfterReboot helper used in a chocolateyinstall.ps1 after the Install-ChocolateyPackage command to create a RETRYpackages.config and scheduled task to install packages from it after reboot (or timeframe). That would (attempt) installing packages sequentially insuring choco.exe doesn't have to run multiple instances. it could probably all be implemented in one line of the scheduled task to do the installs, delete the task, and delete the RETRYpackages.config file.

Assuming theory tested positive, the helper could be a wrapper for Install-ChocolateyPackage that maintainers would use the yet-another-non-existent-but-being-discussed-helper-function instead of Install-ChocolateyPackage that would in turn run Install-ChocolateyPackage and setup the package re-installation attempt upon installation failure.

Of course the "ultimate solution" or alternative "simple solution" would be to just use https://chocolatey.org/packages/choco-persistent-packages for anything that "must" be installed. After configuration, it would just continue to attempt installation of the packages forever until they succeeded. Probably too much effort for the "normal" person, but enterprises could simply script installing https://chocolatey.org/packages/choco-persistent-packages (which is doubtful to ever fail), dump a \ProgramData\Chocolatey\config\persistentpackages.config file in place, and then update that file via GPO or other means as needed in the future.

Just as a side point, for the common use case of MSI 1618 (another install is in progress) - it is better to do a retry loop right at the time that code is received - but double check the chocolatey code in case chocolatey already does this internally.

This is something IMHO Chocolatey should be handling. I think it's already a low priority upgrade on the choco roadmap. The team could probably implement it fairly easily with an opt-in flag to do do the loop or not and how long to wait. Flag could be a feature, an environmental variable, or a switch (probably worst choice).

BUT, with that said...

I did have it in the back of my mind that similar functionality could be implemented via a helper that could be run right before Install-ChocolateyPackage . Some-as-yet-unwritten-(by-me)-unnamed-helper function (probably Check-WindowsInstaller) could simply check the to see if the Windows Installer service is running or not, notify status and loop for some period of time if so (5 minutes? 10 minutes?), exit out if Windows Installer stops or the time limit gets exceeded, and then possibly throw or just let the package fail the Install-ChocolateyPackage command.

Thoughts?

[bcurran3]

I did have it in the back of my mind that similar functionality could be implemented via a helper that could be run right before Install-ChocolateyPackage . Some-as-yet-unwritten-(by-me)-unnamed-helper function (probably Check-WindowsInstaller) could simply check the to see if the Windows Installer service is running or not, notify status and loop for some period of time if so (5 minutes? 10 minutes?), exit out if Windows Installer stops or the time limit gets exceeded, and then possibly throw or just let the package fail the Install-ChocolateyPackage command.

So I did run with this and created https://chocolatey.org/packages/chocolatey-preinstaller-checks.extension as it wasn't really in the same mindset as chocolatey-fastanswers.extension.

I have to look more closely at the Chocolatey scripts to see if I can devise a way to capture the $exitCode and do some decisions based on that.

[DarwinJS]

Windows installer service stays active for quite a while after a package processes. You might want to check for the MSI Inprogress registry key. But be aware that key is only active during the InstallExecute phase - so depending on timing you could still clash so still want to monitor for the exit code indicating an active clash.

I would also advise that retries and retry interval be shorter and configurable. The reason is that if I call chocolatey packages during cloud autoscaling events, each package can't be taking 5 minutes if retires. For instance under AWS cloud formation we use AWS codedeploy. We have only an hour to complete all automation before the nose is deemed failed and an entire deployment fails. For complex stacks there can be a lot of work to do.

[bcurran3]

@DarwinJS Thanks for the feedback!

I've done just under a dozen tests and found each time I installed a program via MSI and kicked off a choco install of a package that when the MSI is done, the pre-check pause generated by the extension for the install of choco package catches it right away on the next loop and exits to install the package. Admittedly my testing has been with small non-complex simple singular programs and nothing complex as a big multi-MSI install.

A timeout in the pause loop is exactly the type of feedback I was hoping to get. I started off with 30 second waits, then experimented with 10, then 15, and then back to 30 seconds as the lesser times created too much "screen noise." I could do just the one "pausing" statement with no updates, but I like loop notifications - makes me feel warm and fuzzy that the computer is telling me it's still doing something and not stuck.

I would also advise that retries and retry interval be shorter and configurable.

I've been considering that since the start. Easy enough to do. My plan is to put v0.0.1 out in the wild first to get it in some sort of general use to see how well it works in OP's environments and then put out v0.0.2 that includes configurable retry and timeout variables. Besides the obvious retry and break times, I'm waiting for any options/feedback to come up that I haven't already thought of as well.

What's your thought on when a pending reboot is found to throw out a 5 second interactive prompt asking if you want to reboot (defaulting to no of course)? ...could go in as a configurable option.

For that matter I could put in an option to cancel the first instance of Chocolatey when stuck in a loop. Again, configurable - the assumption is that it would be useful when manually running a choco install and want to kill whatever's in the background to accomplish your foreground task.

All things to try and play with for v0.0.2, and looking for more suggestions on possibilities.

each package can't be taking 5 minutes if retires.... We have only an hour...

Using your scenario, I'm interested to know your preferences: If my helper delays an install due to Windows Installer being busy, the desired package should get installed when Windows Installer becomes available again and you succeed on your goal of installing the program. If my helper times out (configurable or hard-coded) and allows the install to continue even though the Windows Installer is busy then your program will most likely fail (assuming it's an MSI and impossible to really tell since so many .EXE installers are wrappers for .MSIs) but move on faster to other functions. Which scenario is more desirable to you?

[bcurran3]

BTW: follow-up on handling install failures....

I've been trying and trying and trying to get and handle package error codes and haven't figured out a plausible way. I know what Chocolatey helper scripts generate them, but after each error is parsed to a displayed error message the function gets thrown. When I intercept (alias) and run Install-ChocolateyInstallPackage from my Start-PreinstallChecks, Install-ChocolateyInstallPackage never returns back to Start-PreinstallChecks for further processing. I've experimented running Install-ChocolateyInstallPackage with -ea silently continue and failed, I've tried $test=Install-ChocolateyInstallPackage and some other futile efforts. Basically Install-ChocolateyInstallPackage calls Start-ChocolateyProcessAsAdmin which aborts all further processing after an install and I think returns the package error code only to choco.exe (could be wrong here) and then only returns a 0 or -1 to the shell alter. :( I event tried making sure #exitCode and $exitErrorMessage are global for later use but again a return to my Start-PreInstallChecks script never happens. I'm trying to keep all this inside my helper but it doesn't look possible. I do believe running what I want to do is possible but only if functions are added in to support it in an chocolateyinstall/chocolateyuninstall script...though I may be fooling myself. If I could catch the attention of the Chocolatey devs for some advice, I could quit wasting time spinning my wheels in frustration.

[bcurran3]

NOTE: This really should be broken out into two different discussions since it's about two issues.

• chocolatey-preinstaller-checks.extension 0.0.1 is finished and 0.0.2 new features are being discussed.
• chocolatey-postinstaller-checks.extension doesn't exist yet as I need help finding the interjection point to make it happen. It may not even be possible under current concept.

[bcurran3]

NOTE TO SELF: Seems possible via chocolatey/choco#512 but not sure how to reference the error code (yet).

Mention in Gitter but not response to my inquires: $env:ChocolateyExitCode :(

[ferventcoder]

Running choco.exe (runs both a shim and actual), which causes the pause it appears:

chocolatey-agent:

[bcurran3]

@DarwinJS
configurable time outs and possible aborting in 0.0.2

beta is up (-pre-01) https://github.com/bcurran3/ChocolateyPackages/tree/master/chocolatey-preinstaller-checks.extension

I've done a fair amount of testing under "normal" conditions (100+ packages).

Appreciate it if you looked and tested. :) You'll HAVE to download and install it MANUALLY at the moment....

0.0.1 is still in moderation political purgatory.

[bcurran3]

@DarwinJS

The reason is that if I call chocolatey packages during cloud autoscaling events, each package can't be taking 5 minutes if retires.

Didn't hear back from you, but https://chocolatey.org/packages/chocolatey-preinstaller-checks.extension/0.0.2 is out and I think you'll like the new warn/wait/abort handling.