enhance: [2.4] Reduce segmentManager lock granularity

[bigsheeper]

Use a channel level key lock for segments in segmentManager.

issue: #37633, #37630

pr: #37836

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[codecov]

Codecov Report

Attention: Patch coverage is 87.12871% with 26 lines in your changes missing coverage. Please review.

Project coverage is 80.59%. Comparing base (febed0a) to head (ae800ed).

❗ Current head ae800ed differs from pull request most recent head e7ff7be

Please upload reports for the commit e7ff7be to get more accurate results.

Files with missing lines	Patch %	Lines
internal/datacoord/segment_manager.go	89.18%	13 Missing and 7 partials ⚠️
internal/datacoord/services.go	64.70%	5 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##              2.4   #37869       +/-   ##
===========================================
+ Coverage   70.40%   80.59%   +10.19%     
===========================================
  Files        1074     1073        -1     
  Lines      168099   167716      -383     
===========================================
+ Hits       118353   135179    +16826     
+ Misses      45291    28116    -17175     
+ Partials     4455     4421       -34     

Files with missing lines	Coverage Δ
internal/datacoord/services.go	85.26% <64.70%> (+0.23%)	⬆️
internal/datacoord/segment_manager.go	87.30% <89.18%> (-3.97%)	⬇️

... and 313 files with indirect coverage changes

[czs007]

/approve
/lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bigsheeper, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• internal/datacoord/OWNERS [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[bigsheeper]

/run-cpu-e2e

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[bigsheeper]

rerun ut

[czs007]

/lgtm