[DSIP-3][Performance Improvement] Performance improvement for ds 2.0

[caishunfeng]

Description

As we know, the performance optimization is a long-term challenge that needs to be done consistenly, and we hope DS is more stable and high performance.
So this is a parent issue to track and summarize the optimization issue for ds 2.0 performance.
If you have some performance test and suggestion for ds, welcome to comment it.
if you are interested in performance optimization, welcome to do it together.

Functional

• Master
  • [DS-6694][Master] Reduce the refresh resource interval of LowerWeight #6695
  • [Improvement][MasterServer] improve master scan and handle commands #6849
  • [Feature][MasterServer] reduce db query operation when process instance running #6891
  • [Feature][MasterServer] Add cache manager for Master #6963
• Worker
  • [Bug] [Worker] Worker fakes death when it stop itself fail. #6616
  • [Improvement][Log] optimize task log path to reduce task running time #6665
  • Why worker take more time to handle task when it's busy? #6675
• Zookeeper
  • [Improvement][Server] Change the registry session timeout #6660
• DB
  • [Bug] [MasterServer] With MySQL 8.0, Druid make thread blocked to wait lock when load class  #6483
• Netty
  • [Bug] [Server] Netty client create too many channel and make busy network #6653
• Performance Test
  • Stability Test for DS #6678

[github-actions]

Hi:

• Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to [email protected].

[zhongjiajie]

@caishunfeng Please change title from DSIP to DSIP-3 cause we already have two DSIP issue for now. cc: @CalvinKirs @lenboo could you please add labels DSIP to this issue?

[zhongjiajie]

Hi, @caishunfeng I thinks we should not only add some related PR/close issue for DSIP, but also add what we do and what we latest idea to DSIP. Wanted who would interested on it, looking forward this DSIP 🎉

[caishunfeng]

Hi, @caishunfeng I thinks we should not only add some related PR/close issue for DSIP, but also add what we do and what we latest idea to DSIP. Wanted who would interested on it, looking forward this DSIP tada

OK, I will add the latest plan to it.

[lenboo]

@caishunfeng Thank you very much for your contribution.
I created an issue about optimizing log path.
I think it is also a part of performance optimization and can be associated with this issue
#6665 optimize task log path to reduce task running time

[caishunfeng]

@caishunfeng Thank you very much for your contribution. I created an issue about optimizing log path. I think it is also a part of performance optimization and can be associated with this issue #6665 optimize task log path to reduce task running time

OK, added.