Wrong action chain execution order for remote command executions

[kurzandras]

Problem description

Hello! I am experiencing a problem with the execution order of action chains when I am trying to execute it on multiple machines.
The action chain consisted of multiple actions including various script executions, patches, installations and reboot:

1. Script execution (preinit)
2. Updating packages
3. Installing packages
4. Script execution (setup)
5. Updating packages
6. Reboot
7. Script execution (postscript)

This was scheduled for two clients. Now the problem is that on one of the clients it was completely fine and successful, but on the other client it tried to start with the 4th element instead of the 1st, which of course failed. Please take a look on the histories of the two clients:

Steps to reproduce

1. Create an action chain with multiple type of actions including multiple remote scripts
2. Select multiple systems
3. Execute the action chain

Uyuni version

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : uyuni-server-stable
Name           : Uyuni-Server-release
Version        : 2023.12-230900.210.2.uyuni3
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes (automatically)
Status         : out-of-date (version 2023.10-230900.209.1.uyuni3 installed)
Source package : Uyuni-Server-release-2023.12-230900.210.2.uyuni3.src
Summary        : Uyuni Server
Description    :
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Uyuni proxy version (if used)

No response

Useful logs

venv-salt-minion.log
2024-10-17 01:25:08,565 [salt.state       :327 ][ERROR   ][25089] {'stdout': '/tmp/__salt.tmp.iwvo3oji.sh : Timed out after 600 seconds', 'stderr': '', 'retcode': 1, 'pid': 25424}
2024-10-17 01:25:08,574 [salt.state       :327 ][ERROR   ][25089] {'ret': {'cmd_|-remote_command_|-remote_command_|-script': {'name': 'remote_command', 'changes': {'stdout': '/tmp/__salt.tmp.iwvo3oji.sh : Timed out after 600 seconds', 'stderr': '', 'retcode': 1, 'pid': 25424}, 'result': False, 'comment': "Command 'remote_command' run", '__sls__': 'remotecommands', '__run_num__': 0, 'start_time': '01:15:08.077956', 'duration': 600492.613, '__id__': 'remote_command'}}}

rhn_web_ui.log
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
        at com.suse.manager.reactor.PGEventListener.notify(PGEventListener.java:71) ~[rhn.jar:?]
        at com.suse.salt.netapi.event.AbstractEventStream.notifyListeners(AbstractEventStream.java:64) ~[salt-netapi-clientsalt-netapi-client.jar:?]
        at com.suse.manager.reactor.PGEventStream.lambda$processEvents$6(PGEventStream.java:214) ~[rhn.jar:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
        at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) ~[?:?]
        at java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
        at com.suse.manager.reactor.PGEventStream.processEvents(PGEventStream.java:208) ~[rhn.jar:?]
        at com.suse.manager.reactor.PGEventStream.lambda$notification$1(PGEventStream.java:192) ~[rhn.jar:?]
        at com.redhat.rhn.frontend.events.TransactionHelper.run(TransactionHelper.java:63) ~[rhn.jar:?]
        at com.redhat.rhn.frontend.events.TransactionHelper.handlingTransaction(TransactionHelper.java:48) ~[rhn.jar:?]
        at com.suse.manager.reactor.PGEventStream.lambda$notification$3(PGEventStream.java:191) ~[rhn.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-10-17 01:25:14,369 [salt-event-thread-1] ERROR com.redhat.rhn.frontend.events.TransactionHelper - null
com.suse.manager.reactor.PGEventListenerException: null
        at com.suse.manager.reactor.PGEventListener.lambda$notify$1(PGEventListener.java:80) ~[rhn.jar:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
        at com.suse.manager.reactor.PGEventListener.lambda$notify$2(PGEventListener.java:74) ~[rhn.jar:?]
        at java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411) ~[?:?]
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) ~[?:?]
        at com.suse.manager.reactor.PGEventListener.notify(PGEventListener.java:71) ~[rhn.jar:?]
        at com.suse.salt.netapi.event.AbstractEventStream.notifyListeners(AbstractEventStream.java:64) ~[salt-netapi-clientsalt-netapi-client.jar:?]
        at com.suse.manager.reactor.PGEventStream.lambda$processEvents$6(PGEventStream.java:214) ~[rhn.jar:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
        at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) ~[?:?]
        at java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
        at com.suse.manager.reactor.PGEventStream.processEvents(PGEventStream.java:208) ~[rhn.jar:?]
        at com.suse.manager.reactor.PGEventStream.lambda$notification$1(PGEventStream.java:192) ~[rhn.jar:?]
        at com.redhat.rhn.frontend.events.TransactionHelper.run(TransactionHelper.java:63) ~[rhn.jar:?]
        at com.redhat.rhn.frontend.events.TransactionHelper.handlingTransaction(TransactionHelper.java:48) ~[rhn.jar:?]
        at com.suse.manager.reactor.PGEventStream.lambda$notification$3(PGEventStream.java:191) ~[rhn.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-10-17 01:25:14,378 [salt-event-thread-1] ERROR com.suse.manager.reactor.PGEventStream - Events [802367] were lost

Additional information

No response