Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bundle] Bundle Orchestrator - Batches #3260

Merged
merged 63 commits into from
May 26, 2023
Merged

[Bundle] Bundle Orchestrator - Batches #3260

merged 63 commits into from
May 26, 2023

Conversation

fhibf
Copy link
Contributor

@fhibf fhibf commented Apr 25, 2023 •
edited
Loading

Description

In Bundle Orchestrator Part 1, resources in bundles of type "batch" are processed in parallel.
As soon as the bundle is received, resources are grouped by the respective operation type assigned and then processed in parallel.

Bundle Orchestrator is responsible for orchestrating/aggregating those resources on the fly and sending all them in a single operation to the database.

image

Some important changes made in this PR:

  • New class DataStoreOperationIdentifier: this is a new class that encapsulates the Resource Key and other additional attributes used to identify globally a single resource in the middle of a collection of resources processed by MergeAsync.

  • New Bundle Configuration: I've added a new flag to enable/disable Bundle Orchestrator.

  • BundleHandler: making BundleHandler class thread-safe.

  • Modifying MergeAsync signature: modifying MergeAsync returned data type to use DataStoreOperationIdentifier and better identify the resource submitted in the middle of all resources processed.

  • Bundle Orchestrator Operation ID: I've changed multiple parts of the code to propagate the Bundle Orchestrator Operation ID, to allow the data store layer to recognize that a resource is part of a bundle.

  • New HTTP Header tag: new HTTP header tag to allow customers running bundles sequentially or in parallel.

  • New Bundle validation: new validation to avoid duplicated resources in the same bundle.

Related issues

Addresses [issue #].

Testing

Describe how this change was tested.

FHIR Team Checklist

  • Update the title of the PR to be succinct and less than 65 characters
  • Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
  • Tag the PR with the type of update: Bug, Build, Dependencies, Enhancement, New-Feature or Documentation
  • Tag the PR with Open source, Azure API for FHIR (CosmosDB or common code) or Azure Healthcare APIs (SQL or common code) to specify where this change is intended to be released.
  • CI is green before merge Build Status
  • Review squash-merge requirements

Semver Change (docs)

Patch|Skip|Feature|Breaking (reason)

@fhibf
Batch Orchestrator - Structure and first tests
a642bfe
@fhibf
Fix test
2290cfb
@fhibf
More tests
3058492
@fhibf
Renaming classes. Adding thread-safe logic.
1f75f04
@fhibf
Renamed few existing classes.
5effeaa 
Added more tests.
Split tests by Orchestrator and Operation.
Added more thread-safe validations.
@fhibf
Merge branch 'main' into personal/fhibf/batchOrchestrator
cfe6748
@fhibf
Renaming classes.
82de71d
@fhibf
Merge branch 'personal/fhibf/batchOrchestrator' of https://github.com…
e3f161e 
…/microsoft/fhir-server into personal/fhibf/batchOrchestrator
@fhibf
Adding logic to process requests in a bundle in parallel.
e88b7e5
@fhibf
Adding Bundle Orchestration to startup.
abc5ff3
@fhibf
Adding reference to IFhirDataStore.
7884451
@fhibf
Merge with main.
d7b31df
@fhibf
Adding new dependency to ResourceWrapper.
f0c0392 
Creating ResourceWrapper samples on the flight during tests.
@fhibf
New Bundle Orchestrator changes. New tests. New configuration setup.
ca32922
@fhibf
Adding logic to complete a Bundle Operation.
52a0626 
Adding logic to release resources if they do not return 200 or 201.
@fhibf
Merge with main.
c825397
@fhibf
Adding Bundle Operation ID to the level of FhirController.
b1e2ec9
@fhibf
FhirController changes.
3ddd49f
@fhibf
Propagate Bundle Operation ID between request layers of FHIR.
e0c7d08
@fhibf fhibf added Enhancement Enhancement on existing functionality. Enhancement-Optimization Optimization on existing functionality. Azure API for FHIR Label denotes that the issue or PR is relevant to the Azure API for FHIR Azure Healthcare APIs Label denotes that the issue or PR is relevant to the FHIR service in the Azure Healthcare APIs labels Apr 25, 2023
@fhibf fhibf added this to the S113 milestone Apr 25, 2023
@fhibf fhibf requested a review from a team as a code owner April 25, 2023 21:52
@fhibf
Simplifying the usage of data stores.
050bf15
@fhibf
Moving classes around.
f7fefd9
@fhibf
Supporting new merge async logic associated with Bundle Orchestrator.
5cfe18f
@fhibf
Merge main.
8f063a4
@fhibf
Moving BundleConfiguration to Fhir.Core.
fc645c2 
Making BundleConfiguration a dependency of BundleOrchestrator.
Adding a new configuration to AppSettings allowing BundleOrchestrator to be configurable.
LTA-Thinking
LTA-Thinking reviewed May 24, 2023
View reviewed changes
Copy link
Collaborator

@LTA-Thinking LTA-Thinking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been through the non-test files.

...crosoft.Health.Fhir.Core.UnitTests/Features/Persistence/DataStoreOperationIdentifierTests.cs Show resolved Hide resolved
src/Microsoft.Health.Fhir.Tests.Common/Microsoft.Health.Fhir.Tests.Common.csproj Outdated Show resolved Hide resolved
src/Microsoft.Health.Fhir.Shared.Api/Features/Resources/Bundle/BundleHandler.cs Show resolved Hide resolved
src/Microsoft.Health.Fhir.Shared.Api/Features/Resources/Bundle/BundleHandler.cs Outdated Show resolved Hide resolved
src/Microsoft.Health.Fhir.Shared.Api/Features/Resources/Bundle/BundleHandlerStatistics.cs Outdated Show resolved Hide resolved
...icrosoft.Health.Fhir.Shared.Api/Features/Resources/Bundle/BundleHandlerParallelOperations.cs Outdated Show resolved Hide resolved
...icrosoft.Health.Fhir.Shared.Api/Features/Resources/Bundle/BundleHandlerParallelOperations.cs Show resolved Hide resolved
src/Microsoft.Health.Fhir.CosmosDb/Features/Storage/CosmosFhirDataStore.cs Outdated Show resolved Hide resolved
...Microsoft.Health.Fhir.Core/Features/Persistence/Orchestration/BundleOrchestratorOperation.cs Outdated Show resolved Hide resolved
...t.Health.Fhir.Core/Features/Persistence/Orchestration/BundleOrchestratorOperationResource.cs Outdated Show resolved Hide resolved
SergeyGaluzo
SergeyGaluzo approved these changes May 26, 2023
View reviewed changes
Copy link
Contributor

@SergeyGaluzo SergeyGaluzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@fhibf fhibf merged commit 0bdea5e into main May 26, 2023
@fhibf fhibf deleted the personal/fhibf/batchOrchestrator branch May 26, 2023 23:06
fhibf added a commit that referenced this pull request Jun 5, 2023
@fhibf
Revert "[Bundle] Bundle Orchestrator - Batches (#3260)"
62e16d2 
This reverts commit 0bdea5e.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abiisnn abiisnn abiisnn left review comments

@SergeyGaluzo SergeyGaluzo SergeyGaluzo approved these changes

@LTA-Thinking LTA-Thinking LTA-Thinking approved these changes

Assignees
No one assigned
Labels
Azure API for FHIR Label denotes that the issue or PR is relevant to the Azure API for FHIR Azure Healthcare APIs Label denotes that the issue or PR is relevant to the FHIR service in the Azure Healthcare APIs Enhancement Enhancement on existing functionality. Enhancement-Optimization Optimization on existing functionality.
Projects
None yet
Milestone
S114
Development

Successfully merging this pull request may close these issues.
4 participants
@fhibf @SergeyGaluzo @LTA-Thinking @abiisnn