[Feature][Task Plugin] RemoteShell Task Plugin

[jieguangzhou]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

A new task plugin, this allows us to run tasks on machines that do not have worker installed

Use case

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[jieguangzhou]

It's similar to this, but here it runs as a component #12736

[EricGao888]

@jieguangzhou Great! Looking forward to this feature.

Also I have a couple of questions:

1. How to monitor the status of tasks after DS submits them to remote machine?
2. How to stop / kill / restart ssh tasks?
3. How to manage resource files of ssh tasks?

BTW, I see you label this feature with 4.0.0-alpha, why not 3.2.0?

[jieguangzhou]

@jieguangzhou Great! Looking forward to this feature.

Also I have a couple of questions:

1. How to monitor the status of tasks after DS submits them to remote machine?
2. How to stop / kill / restart ssh tasks?
3. How to manage resource files of ssh tasks?

BTW, I see you label this feature with 4.0.0-alpha, why not 3.2.0?

I want to use Apache MINA SSHD to run commands on the remote machine.

1. I plan to give a special identifier to the program when it runs, such as "dolphinscheduler-remoteshell". Later, to monitor the program, we just need to check if there are processes with this identifier (for example, using the command "ps -ef | grep dolphinscheduler-remoteshell-taskid"). Additionally, the program's logs will be stored in a specific path on the remote machine, and the task status will be monitored by combining the process status with the logs.
2. For process management, we can use Mina-sshd to send the corresponding commands to the server, such as "kill -9".
3. I haven't dealt with resource files yet because I assumed from the beginning that the program would be running on a remote server that already has all the required resources, such as environment, data, and resource files.

[jieguangzhou]

@jieguangzhou Great! Looking forward to this feature.

Also I have a couple of questions:

1. How to monitor the status of tasks after DS submits them to remote machine?
2. How to stop / kill / restart ssh tasks?
3. How to manage resource files of ssh tasks?

BTW, I see you label this feature with 4.0.0-alpha, why not 3.2.0?

I have already updated the label to 3.2.

[Radeity]

3. I haven't dealt with resource files yet because I assumed from the beginning that the program would be running on a remote server that already has all the required resources, such as environment, data, and resource files.

It seems like we can also use sshd-scp provided by Apache MINA SSHD to send resources files managed by DS to remote machine.