Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: restart interrupted data flows #4612

Merged

Conversation

ndr-brt
Copy link
Member

@ndr-brt ndr-brt commented Nov 8, 2024

What this PR changes/adds

Restarts long running transfers after a data-plane restart

Why it does that

dataplane should be able to recover the transfers after a reboot

Further notes

  • this only provides the feature in a single data-plane instance environment, clustered environment will be supported in subsequent PRs
  • added @PostgreSQL version of E2E tests for streaming. also because this scenario, that provides for the reboot is only testable with a persistent store and not in memory.
  • added a closeAll method on TransferService that will shut down all the transfers during the shut down phase.
  • implemented the < operator, I was actually surprised that it didn't already existed. this was necessary, I will provide an issue/PR to implement also the greater than.

the restartFlows method will in the future called in a different way as we're reasoning about the startup phases

Linked Issue(s)

Part of #4440

Please be sure to take a look at the contributing guidelines and our etiquette for pull requests.

@ndr-brt ndr-brt added enhancement New feature or request dpf Feature related to the Data Plane Framework labels Nov 8, 2024
@jimmarino
Copy link
Contributor

Can we get a DR first on the startup processors? I'd like to have this information documented and also have a quick discussion if this should be generalized. A common pattern is to be able to register a general callback during bootstrap for when a runtime enters a ready state.

@ndr-brt ndr-brt force-pushed the 4440-restart-interrupted-data-flows branch from 842d0f3 to a2b904f Compare November 11, 2024 10:31
@ndr-brt ndr-brt force-pushed the 4440-restart-interrupted-data-flows branch from 9bcb07b to 8ac5615 Compare November 13, 2024 15:27
@ndr-brt ndr-brt merged commit df0569e into eclipse-edc:main Nov 18, 2024
21 checks passed
@ndr-brt ndr-brt deleted the 4440-restart-interrupted-data-flows branch November 18, 2024 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dpf Feature related to the Data Plane Framework enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants