"local-shell-script" does not work on read only file systems

[anrajme]

SUMMARY

While using the action runner type "local-shell-script", the action always try to set execute permission for the action script before the execution.

For example, action runner logs for the core.sendmail action

2022-03-09 07:02:06,866 INFO [-] Executing action via LocalRunner: dcc6c28d-5674-4cc2-9174-ca6c1e7b23a3
2022-03-09 07:02:06,866 INFO [-] [Action info] name: sendmail, Id: 622850eead44b087ebe77063, command: chmod +x /opt/stackstorm/packs/core/actions/send_mail/send_mail ; sudo -E -H -u stanley -- bash -c '/opt/stackstorm/packs/core/actions/send_mail/send_mail None stanley [email protected] '"'"'validate st2 local-shell-script runner'"'"' 1 text/html '"'"'validate st2 local-shell-script runner content'"'"' '"'"''"'"'', user: stanley, sudo: False

This "chmod +x " operation cause failure when you are using stackstorm-ha helm chart based deployment with an immutable file system for packs.

https://docs.stackstorm.com/install/k8s_ha.html#method-1-st2packs-images-the-default

For example, my custom pack deployed via the st2pack image method fails to run the action since the "chmod +x" instruction cannot be executed on a read-only file system.

{
  "failed": true,
  "succeeded": false,
  "return_code": 1,
  "stdout": "",
  "stderr": "chmod: changing permissions of '/opt/stackstorm/packs/workflow/actions/verify_env.sh': Read-only file system"
}

Logs:

2022-03-08 13:06:39,736 INFO [-] Executing action via LocalRunner: a2bdfba5-682b-47b2-9485-4241b94501cf
2022-03-08 13:06:39,737 INFO [-] [Action info] name: verify_env, Id: 622754dfb56ccf38de50a292, command: chmod +x /opt/stackstorm/packs/workflow/actions/verify_env.sh ; sudo -E -H -u stanley -- bash -c '/opt/stackstorm/packs/workflow/actions/verify_env.sh '"'"'MY_ARGUMENTS_HERE'"'"'', user: stanley, sudo: False
2022-03-08 13:06:39,748 INFO [-] [622754de749089f0a983f6c3] Action execution "622754dfb56ccf38de50a293" for task "verify_env" is updated and in "running" state.
2022-03-08 13:06:39,848 INFO [-] [622754de749089f0a983f6c3] Action execution "622754dfb56ccf38de50a293" for task "verify_env" is updated and in "failed" state.
2022-03-08 13:06:39,855 INFO [-] Found 0 rules defined for trigger core.st2.generic.actiontrigger
2022-03-08 13:06:39,856 INFO [-] No matching rules found for trigger instance 622754dfe3850ebc7fd8d8c8.

I guess it's failing because of the condition mentioned here below for the local shell script runner.

https://github.com/StackStorm/st2/blob/master/contrib/runners/local_runner/local_runner/base.py#L121

STACKSTORM VERSION

root@m1-staging-st2actionrunner-6664d699c-55z5t:/opt/stackstorm# st2 --version
st2 3.6.0, on Python 3.6.9

OS, environment, install method

stackstorm-ha helm deployed k8s environment running on EKS cluster

Steps to reproduce the problem

Follow the method mentioned here to deploy a custom st2 pack image and try to execute the action with runner type as "local-shell-script"

https://docs.stackstorm.com/install/k8s_ha.html#method-1-st2packs-images-the-default

Expected Results

The action executes fine without a "chmod: changing permissions of '/opt/stackstorm/packs/custom-pack/actions/custom-script.sh': Read-only file system" failure and return the results.

Actual Results

{
  "failed": true,
  "succeeded": false,
  "return_code": 1,
  "stdout": "",
  "stderr": "chmod: changing permissions of '/opt/stackstorm/packs/workflow/actions/verify_env.sh': Read-only file system"
}

Thanks!

[cognifloyd]

This may present itself in HA, but the bug cannot be addressed in the helm chart. The action runner needs to be modified to support read only file systems and skip chmod in that case.

[arm4b]

How the scripts would run if they had no executable bit before and chmod is required?

@anrajme does it still complain if you make the scripts executable in advance?

[anrajme]

@cognifloyd - absolutely..! @armab - yeah, tried that approach. Though the scripts already have exec permission enabled, the runner tries to "chmod +x" every time it executes.

Maybe we can add a condition to the action runner code to verify the permission and try chmod only if the exec bits are not enabled. In that case, we can set the file permission in advance before creating st2 pack image and ensure this step will be passed. I'm still learning, wondering whether this is the right piece of code for that.

https://github.com/StackStorm/st2/blob/master/contrib/runners/local_runner/local_runner/base.py#L121

In the interim, I'm planning to switch to a python-script runner to get things going.

cheers!

[arm4b]

@anrajme For sure, the condition to skip chmod +x when the executable bit is already set makes perfect sense 👍

[jk464]

Just hit this myself.

Although I do question why we'd want to run chmod +x anyway. Surely the pack content should be static, and if a file has missing exec permissions thats a bug in the pack and should be fixed, rather than using this workaround to fix, and thus cover up, the issue?

Unless of course I am missing the situation, where there may be a file being ran, thats user provided and not in the pack itself, and thus needs permissions set at run time? Then again, if the file system is read-only - only files present in the installed pack could be used, and there wouldn't be a way for the user provide this theoretical file..