refactor(source): remove chunk splitting logic in apply_rate_limit

[stdrc]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

After #19698, we can expect chunks yielded to SourceExecutors to be smaller than SourceCtrlOpts::chunk_size, so we don't need to split chunks in apply_rate_limit (used by FetchExecutor, FsSourceExecutor, SourceBackfillExecutor and SourceExecutor) anymore.

This PR removes the chunk splitting logic from apply_rate_limit.

Checklist

• I have written necessary rustdoc comments.
• I have added necessary unit tests and integration tests.
• I have added test labels as necessary.
• I have added fuzzing tests or opened an issue to track them.
• My PR contains breaking changes.
• My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
• My PR contains critical fixes that are necessary to be merged into the latest release.

Documentation

• My PR needs documentation updates.

Release note

[stdrc]

• refactor(source): remove chunk splitting logic in apply_rate_limit #19826 👈 (View in Graphite)
• refactor(source): respect max chunk size in source connector parser #19698 : 1 other dependent PR (#19863 )
• refactor(cdc source): merge cdc heartbeat chunk builder & data chunk builder #19671
• refactor(connector): re-arrange the connector::parser module for better readibility #19697
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[xxchan]

rubber stamp

[tabVersion]

rest LGTM

[MrCroxx]

Rest LGTM.

[MrCroxx]

Can we remove it since there is no limit for burst?

[stdrc]

I slightly prefer keeping this, because if we don't have this, we may have difficulty to decide between chunk.capacity() and chunk.cardinality() later.