fix: fix performance issues due to contention in the integrations table when publishing a large number of contracts (> 20) per request, in parallel #654
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For a particular customer who was publishing 19 contracts at a time with requests that overlapped, we were seeing that the lock on the integrations table was causing each subsequent request to take longer and longer as each waited for the lock on the integrations table to be released. The problem has been fixed by updating the integration rows all at once at the end of the request, rather than updating them one at a time when each pact gets saved.
Problematic way:
create/update pact 1
update integration 1
create/update pact 2
update integration 2
create/update pact 3
update integration 3
create/update pact 4
update integration 4
New way:
create/update pact 1
create/update pact 2
create/update pact 3
create/update pact 4
update integrations 1, 2, 3, 4
The code uses a pub/sub pattern using the Wisper library to update the
contract_data_updated_at
field on the integration when a consumer contract or a provider contract is published. The subscribed functions get invoked synchronously - the pub/sub is just a way to decouple the repositories so that the pacts/verifications repositories don't need to know about calling the integrations respository. It's possible that this layer of indirection is overkill 🤷🏽The problem with this pattern is that it was causing the integration to be updated every time the pact was published, and because we now have an endpoint that allows the publication of multiple pacts at once, this was leading to the problem described above.
The new code only subscribes the integration listener to the individual resource endpoints (PUT pact, and POST verification), and manually batch updates the integrations in one go in the Contract Service, which was designed to handle the batch contract publications.