Fix linux pack actions For CentOS 8 #5035

winem · 2020-09-04T21:51:42Z

Part of #4999

This PR addresses the actions that fail on CentOS8 but work on other operating systems and distributions and writes the output of the load_diag actionchain to a json file which is parsable via jq or other tools and therefore hopefully a bit more useful to the users.

I did not add any tests yet. I would do this in another PR or give others the chance to add tests.

…thout python (only python3 and python2)

contrib/linux/actions/checks/check_processes.py

arm4b

Thanks for trying fixing this EL8 issue!

Per comments in this PR, we can't rely on a python 2 env only considering active development and aim towards the Python 2 Deprecation.

There might be other ways to explore to fix the pack.

arm4b · 2020-09-17T14:02:55Z

BTW is this an issue only on EL8 or in any python3 environment, including Ubuntu Bionic?

winem · 2020-09-17T15:43:58Z

This issue applies to CentOS8 only. Ubuntu 18.04 has python in the path which is linked to one of the installed Python versions. I don't know for sure which one but I think it's the Python3.x

That's why a shebang like #!/usr/bin/env python works on the supported Ubuntu distributions.

arm4b · 2020-09-17T21:33:01Z

Can you elaborate more on the root cause of the issue? Is just #!/usr/bin/env python failing to find the python binary in EL8 system or something else?
If so, can we hardcode it and use the st2 py one, like #!/opt/stackstorm/st2/bin/python? This would also use the st2 virtualenv with all its installed dependencies.

winem · 2020-09-18T09:01:51Z

Can you elaborate more on the root cause of the issue? Is just #!/usr/bin/env python failing to find the python binary in EL8 system or something else?
If so, can we hardcode it and use the st2 py one, like #!/opt/stackstorm/st2/bin/python? This would also use the st2 virtualenv with all its installed dependencies.

Exactly, that should to it and seems to be an optimal solution. I'll test this tonight!

CHANGELOG.rst

…n checks

winem · 2020-09-19T19:43:43Z

Can you elaborate more on the root cause of the issue? Is just #!/usr/bin/env python failing to find the python binary in EL8 system or something else?
If so, can we hardcode it and use the st2 py one, like #!/opt/stackstorm/st2/bin/python? This would also use the st2 virtualenv with all its installed dependencies.

Thanks, this works like a charm and it's finally a solution that satisfies me! Latest commit includes the change.

…fix-linux-pack-actions

arm4b · 2020-09-22T20:05:02Z

contrib/linux/actions/workflows/diag_loadavg.yaml

@@ -47,5 +43,5 @@
      name: "dump_results"
      ref: "core.local"
      parameters:
-        cmd: "echo \"ST2 Workflow\tdiag_loadavg:\t{{__results}}\" >> /tmp/diag_loadavg && echo 'Output written to file /tmp/diag_loadavg'"
+        cmd: "echo \"{{__results}}\" | python2 -c \"import json; import sys; from datetime import datetime; diag_data=eval(sys.stdin.read()); diag_data['timestamp'] = str(datetime.now()); print(json.dumps(diag_data))\" >> /tmp/diag_loadavg.json && echo 'Output written to file /tmp/diag_loadavg.json'"


Looks like this is the only place with the hardcoded python2 now.
Is it possible to do something with that so we don't rely on a specific version?

Besides of that, looks good.

Yeah, any reason why we need to use python2 here? I believe that line should also work fine with Python 3 (aka just using python system binary - should work if it's either python 2 or python 3).

I also wonder if there is a better way which doesn't rely on eval() which can be unsafe, especially in this context when we are passing 3rd party data to it.

I don't see any issues regarding py3 and it should work as well (I can put that on my list again).

Let's also tackle the eval. The safe alternative without a bunch of JSON processing on stdin is ast.literal_eval: https://docs.python.org/3/library/ast.html#ast.literal_eval

Would this be fine for everyone? Last resort would be some manual parsing & processing with the JSON library.

ast.literal_eval is awesome (and quite safe)! go for it.

cognifloyd

It doesn't look like the black reformat will interfere with this PR much. (hooray!) Master still needs to be merged in though (looks pretty minor).

cognifloyd · 2021-03-07T06:33:58Z

contrib/linux/actions/workflows/diag_loadavg.yaml

@@ -47,5 +43,5 @@
      name: "dump_results"
      ref: "core.local"
      parameters:
-        cmd: "echo \"ST2 Workflow\tdiag_loadavg:\t{{__results}}\" >> /tmp/diag_loadavg && echo 'Output written to file /tmp/diag_loadavg'"
+        cmd: "echo \"{{__results}}\" | python2 -c \"import json; import sys; from datetime import datetime; diag_data=eval(sys.stdin.read()); diag_data['timestamp'] = str(datetime.now()); print(json.dumps(diag_data))\" >> /tmp/diag_loadavg.json && echo 'Output written to file /tmp/diag_loadavg.json'"


ast.literal_eval is awesome (and quite safe)! go for it.

CLAassistant · 2021-09-06T14:18:46Z

All committers have signed the CLA.

cognifloyd · 2021-10-05T21:35:02Z

I merged in master. Hopefully we can get this bugfix over the hump quickly. If not in 3.6.0, then in 3.7.0.

arm4b

There's a need to update the pack version following the changes in this PR:

st2/contrib/linux/pack.yaml

Line 19 in a82b1be

version : 1.0.2

otherwise looks good.

winem added 4 commits September 4, 2020 22:48

Print the diag_loadavg output as plain json to the output file

6eba49d

Use python2 for the the linux checks to mitigate issues on CentOS8 wi…

f38ccda

…thout python (only python3 and python2)

Add changelog entry for the centos8 linux pack fixes

9139fc0

Add note about lsof to be installed manually on centos8

473a941

pull-request-size bot added the size/S PR that changes 10-29 lines. Very easy to review. label Sep 4, 2020

Merge branch 'master' into fix-linux-pack-actions

7be79f6

arm4b reviewed Sep 8, 2020

View reviewed changes

contrib/linux/actions/checks/check_processes.py Outdated Show resolved Hide resolved

arm4b suggested changes Sep 17, 2020

View reviewed changes

arm4b added bug OS support Support/issues/PRs on a specific OS labels Sep 17, 2020

arm4b reviewed Sep 18, 2020

View reviewed changes

CHANGELOG.rst Outdated Show resolved Hide resolved

Use the st2 py instead of the system python installation on the pytho…

50f2774

…n checks

winem added 4 commits September 19, 2020 21:51

Use st2 python for the linux.service action, too

fb4d4dc

Update changelog

cda720d

Merge branch 'fix-linux-pack-actions' of pgithub.com:/winem/st2 into …

88222d1

…fix-linux-pack-actions

Update changelog format

cbf523c

winem mentioned this pull request Sep 19, 2020

ImportError: No module named st2common.util.shell when run linux.service #5004

Open

arm4b reviewed Sep 22, 2020

View reviewed changes

cognifloyd reviewed Mar 7, 2021

View reviewed changes

Merge branch 'master' into fix-linux-pack-actions

1c12531

cognifloyd added the status:needs cla A manual tag to note PRs that are otherwise ready, but the CLA bot says the CLA has not been signed. label Oct 5, 2021

cognifloyd added this to the 3.6.0 milestone Oct 7, 2021

Merge branch 'master' into fix-linux-pack-actions

b25cc4f

cognifloyd removed the status:needs cla A manual tag to note PRs that are otherwise ready, but the CLA bot says the CLA has not been signed. label Apr 1, 2022

cognifloyd modified the milestones: 3.6.0, 3.7.0 Apr 1, 2022

cognifloyd added 3 commits April 1, 2022 13:18

Merge branch 'master' into fix-linux-pack-actions

b756326

Use python3 and ast.literal_eval

749c8bf

drop whitespace at eol

a82b1be

cognifloyd approved these changes Apr 1, 2022

View reviewed changes

cognifloyd requested a review from arm4b April 1, 2022 18:47

arm4b approved these changes Apr 1, 2022

View reviewed changes

cognifloyd added 2 commits April 1, 2022 19:41

Merge branch 'master' into fix-linux-pack-actions

f8f7951

bump linux pack version for EL 8 suport

cb4f9bb

cognifloyd enabled auto-merge April 2, 2022 00:43

cognifloyd merged commit 2a9a3dc into StackStorm:master Apr 2, 2022

arm4b mentioned this pull request Sep 15, 2023

action linux.service or linux.check_processes cannot be executed on remote host #5983

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix linux pack actions For CentOS 8 #5035

Fix linux pack actions For CentOS 8 #5035

winem commented Sep 4, 2020 •

edited by cognifloyd

Loading

arm4b left a comment

arm4b commented Sep 17, 2020

winem commented Sep 17, 2020 •

edited

Loading

arm4b commented Sep 17, 2020 •

edited

Loading

winem commented Sep 18, 2020

winem commented Sep 19, 2020

arm4b Sep 22, 2020 •

edited

Loading

Kami Feb 22, 2021

Kami Feb 22, 2021

winem Feb 23, 2021

cognifloyd Mar 7, 2021

This comment was marked as outdated.

cognifloyd left a comment

cognifloyd Mar 7, 2021

CLAassistant commented Sep 6, 2021 •

edited

Loading

cognifloyd commented Oct 5, 2021

arm4b left a comment •

edited

Loading

Fix linux pack actions For CentOS 8 #5035

Fix linux pack actions For CentOS 8 #5035

Conversation

winem commented Sep 4, 2020 • edited by cognifloyd Loading

arm4b left a comment

Choose a reason for hiding this comment

arm4b commented Sep 17, 2020

winem commented Sep 17, 2020 • edited Loading

arm4b commented Sep 17, 2020 • edited Loading

winem commented Sep 18, 2020

winem commented Sep 19, 2020

arm4b Sep 22, 2020 • edited Loading

Choose a reason for hiding this comment

Kami Feb 22, 2021

Choose a reason for hiding this comment

Kami Feb 22, 2021

Choose a reason for hiding this comment

winem Feb 23, 2021

Choose a reason for hiding this comment

cognifloyd Mar 7, 2021

Choose a reason for hiding this comment

This comment was marked as outdated.

cognifloyd left a comment

Choose a reason for hiding this comment

cognifloyd Mar 7, 2021

Choose a reason for hiding this comment

CLAassistant commented Sep 6, 2021 • edited Loading

cognifloyd commented Oct 5, 2021

arm4b left a comment • edited Loading

Choose a reason for hiding this comment

winem commented Sep 4, 2020 •

edited by cognifloyd

Loading

winem commented Sep 17, 2020 •

edited

Loading

arm4b commented Sep 17, 2020 •

edited

Loading

arm4b Sep 22, 2020 •

edited

Loading

CLAassistant commented Sep 6, 2021 •

edited

Loading

arm4b left a comment •

edited

Loading