Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Closes https://linear.app/ogp/issue/ISOM-833/benchmark-impact-of-local-git-diff-on-cpu-usage
Local diff was implemented a while back but kept behind a feature flag.
Feature flag has not been enabled in production due to concerns about performance especially for larger diffs.
Solution
Understanding bottlenecks via Datadog
Everything below is in the context of motac2 with ~444 files changed in one RR
Trace for unoptimised implementation
Significant bottlenecks (in order of priority):
getFilesChanged
27s / 40s or ~66%getLatestCommitOfPath
for every single file changedUsers
for every commit to getemail
(@timotheeg had already fixed this for the GitHub diff flow), actually doesn't add much to latencyOptimisations
getFilesChanged
to callgit diff-tree -r --name-only master..staging
getCommitsBetweenMasterAndStaging
oncefilenameToLogMap
. At the same time, get uniqueuserIds
from commit messages to dedupe queriesuserIdToUserMap
Final flame graph after optimisations:
Bottleneck is now the many parallel reads on EFS. Out of scope for this PR and its also current behaviour for GitHub flow.
TLDR: reads are needed to get the permalink from frontmatter, which is then used to construct the
stagingUrl
Screen.Recording.2024-04-25.at.10.30.55.PM.mov
Breaking Changes
Tests
Copied over tests from #1172
is_local_diff_enabled
for account being tested https://app.growthbook.io/features/is_local_diff_enabled