fix(bpdm-orchestrator): introduced pagination for handling pending and retention timeouts

[SujitMBRDI]

Description

This pull request introduces pagination for handling pending and retention timeouts in the orchestrator. By processing tasks in manageable batches, the changes prevent out-of-memory errors during the execution of large volumes of tasks, improving the system's stability and efficiency.

Pre-review checks

Please ensure to do as many of the following checks as possible, before asking for committer review:

• DEPENDENCIES are up-to-date. Dash license tool. Committers can open IP issues for restricted libs.
• Copyright and license header are present on all affected files

[nicoprow]

The logic looks sound to me but I think using such an unspecifc public function in the golden record task service in which you can basically place in any fetch and process function to be executed seems to be quite over the top. Especially when we consider that the TimeoutProcessBatchService does have two very similar methods that just differentiate in between which they give delegates as parameters to that function in the golden record task service.

Why not just create two separate functions in the golden record task service, one for fetching and processing pending timeouts, one for fetching and processing retention timeouts. If you then want to reduce duplicate code you could create a private function for batch processing these functions.

[nicoprow]

Details for my previous comments in the code

[SujitMBRDI]

The logic looks sound to me but I think using such an unspecifc public function in the golden record task service in which you can basically place in any fetch and process function to be executed seems to be quite over the top. Especially when we consider that the TimeoutProcessBatchService does have two very similar methods that just differentiate in between which they give delegates as parameters to that function in the golden record task service.

Why not just create two separate functions in the golden record task service, one for fetching and processing pending timeouts, one for fetching and processing retention timeouts. If you then want to reduce duplicate code you could create a private function for batch processing these functions.

Yeah, will refactor code accordingly.

[SujitMBRDI]

Details for my previous comments in the code

@nicoprow, i have adapted changes as suggested.

[nicoprow]

Looks good to me. Minor suggestions in the comments