Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: job cancellation capability for IRS & utility class for test data generation #716

Conversation

ds-pweick
Copy link
Contributor

Job cancellation in IRS using Java interrupts

This PR addresses issue #411.

  • AASTransferProcessManager

    • When a new thread is taken from the thread pool to execute a new transfer process, instead of using execute, the code now calls submit for the created Runnable. This returns a Future, which is then stored in a ConcurrentHashMap. The map's keys are the uuid strings of the transfer processes, thus making it possible to associate a transfer process id with the task it represents.

    • To make use of this, there is now a cancelRequest method which retrieves a Future from the map using a transfer process uuid string argument. Then, cancel(true) is called on that Future, which sets the interrupt flag for the thread executing the transfer process. Afterwards, the entry for that transfer process is removed from the map.

    • Before initiating a request, this interrupt flag is checked. If it is set, the transfer process is not started and a TransferInitiateResponse is returned with a newly added ResponseStatus of NOT_STARTED_JOB_CANCELLED.

    • In case of the unhappy path, where the map somehow doesn't contain an entry for a transfer process or something goes wrong when setting the interrupt flag, JobExceptions are created using a newly added constructor. This constructor allows for directly setting the error detail.

  • JobOrchestrator

    • A cancelJob method has been added which is called when the IRS receives a job cancellation request. It uses a passed MultiTransferJob argument to get all transfer process ids for the job that is to be cancelled and iterates over the collection of ids calling AASTransferProcessManager's cancelRequest method. In each iteration, if cancelRequest throws a JobException, the method evaluates what happened based on the error detail that was specified when creating the Exception in cancelRequest. This detail is then used to construct an error message, which is uploaded to the job store using markJobInError.
  • BaseJobStore's writeLock now reliably triggers an InterruptedException, which allows to cancel any change to a job from one of the interrupted threads running the transfer processes.

  • IrsItemGraphQueryService's cancelJobById now calls cancelJob in JobOrchestrator.

  • Unit tests for AASTransferProcessManager and JobOrchestrator have been modified accordingly.

  • IrsWireMockIntegrationTest now has a method to test IRS job cancellation. The test uses the utility described below.

Utility for test data generation

  • This PR contains a new utility class called SingleLevelTestDataGenerator which can be used to generate single-level test data for a (theoretically, but practically infeasible) arbitrary recursion depth. The utility allows for choosing the number of relationships that each asset holds, if any at all. It uses another class which parses data needed for generation from a JSON file.

ds-pweick and others added 6 commits June 10, 2024 08:46
…jobs using Java interrupts

- Added a job cancellation which works by storing Futures representing transfer processes and calling cancel(true) on all transfer processes belonging to a job

- Each transfer process thread can then check its interrupt flag and will return from its current operation if interrupted

- That way, cancellation should be performed faster than previously
- Generator utility for single-level models testdata that uses definitions in a json file to generate a hierarchy of single-level assets
- Added integration test for IRS job cancellation capability that uses the utility
@ds-pweick ds-pweick marked this pull request as draft June 19, 2024 14:50
@ds-pweick ds-pweick marked this pull request as ready for review July 10, 2024 14:04
@ds-jhartmann
Copy link
Contributor

Closing this PR since we cannot merge this at the moment and will not add this within consortia phase.
Might re-visit this topic in the future

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants