fix(gatsby-source-drupal): cache backlink records #33444

KyleAMathews · 2021-10-05T20:19:54Z

Write the in-memory lookup map to the cache after a successful run of "sourceNodes" and reload it for use in subsequent builds.

This fixes an issue where backlinks aren't preserved properly for incremental node updates with a warm cache.

Fixes #33267

…fore filtering (#33181) (#33228) * fix(gatsby-source-drupal): check relationships type exists on node before filtering * Update packages/gatsby-source-drupal/src/utils.js Co-authored-by: Dustin Schau <[email protected]> * format Co-authored-by: Dustin Schau <[email protected]> (cherry picked from commit d4f8355) Co-authored-by: Kyle Mathews <[email protected]>

…x-pr

…l-inc-build-backlink-fix-pr

…fork

wardpeet · 2021-10-25T19:06:15Z

packages/gatsby-source-drupal/src/utils.js

+    backRefsNamesLookup = new Map(JSON.parse(backRefsNamesLookupStr))
+  }
+
+  if (referencedNodesLookupStr) {
+    referencedNodesLookup = new Map(JSON.parse(referencedNodesLookupStr))


I wonder if this can become very memory expensive. Does this run on PSU without problems? If not we should look into http://ndjson.org/

I tested it on a smaller site but good point — lemme test how expensive this is for a PSU-sized site

rss increases about 400mb while stringifying the two maps into a totoal of ~85 million characters. Execution time is ~600ms.

I think reasonable enough given how big PSU is — they're using ~6.5gb already at this point so this is a temp 15% bump in memory use that'll be GCed fairly soon after that.

GatsbyJS Bot and others added 6 commits September 17, 2021 17:52

Preserve backlink/reference lookup tables in cache

0c10243

Merge branch 'master' into gatsby-source-drupal-inc-build-backlink-fi…

8438e7a

…x-pr

Merge remote-tracking branch 'origin/master' into gatsby-source-drupa…

c49dee8

…l-inc-build-backlink-fix-pr

test

80befd5

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Oct 5, 2021

fix bad merge

f7911d7

KyleAMathews added topic: source-drupal Related to Gatsby's integration with Drupal type: bug An issue or pull request relating to a bug in Gatsby and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Oct 5, 2021

KyleAMathews and others added 5 commits October 5, 2021 13:29

try try again

c7db662

Improve log

9fd8df8

Merge remote-tracking branch 'upstream/master' into kyle-s-fork

9b7e79b

Merge branch 'kyle-s-fork' of github.com:gatsbyjs/gatsby into kyle-s-…

73b7b9d

…fork

Get working with tests

ec1dbd4

KyleAMathews added a commit that referenced this pull request Oct 25, 2021

cache backlink records #33444

3fa6041

wardpeet reviewed Oct 25, 2021

View reviewed changes

wardpeet approved these changes Oct 25, 2021

View reviewed changes

KyleAMathews merged commit 0def3ac into master Oct 25, 2021

KyleAMathews deleted the kyle-s-fork branch October 25, 2021 19:56

gatsbybot mentioned this pull request Oct 27, 2021

fix(gatsby-source-drupal): cache backlink records (#33444) #33701

Merged

This was referenced Apr 15, 2024

[Snyk] Fix for 1 vulnerabilities Xtuden-com/gatsby#5665

Open

[Snyk] Fix for 1 vulnerabilities Xtuden-com/gatsby#5666

Open

[Snyk] Fix for 1 vulnerabilities Xtuden-com/gatsby#5667

Open

[Snyk] Fix for 1 vulnerabilities Xtuden-com/gatsby#5668

Open

MaxMood96 mentioned this pull request Apr 15, 2024

[Snyk] Fix for 1 vulnerabilities MaxMood96/gatsby#1502

Open

This was referenced Apr 15, 2024

[Snyk] Security upgrade gatsby-plugin-sharp from 2.14.4 to 3.15.0 ajesse11x/gatsby#2667

Open

[Snyk] Fix for 1 vulnerabilities ajesse11x/gatsby#2668

Open

This was referenced Apr 15, 2024

[Snyk] Fix for 1 vulnerabilities MaxMood96/gatsby#1503

Open

[Snyk] Fix for 1 vulnerabilities MaxMood96/gatsby#1504

Open

This was referenced Apr 15, 2024

[Snyk] Fix for 1 vulnerabilities ajesse11x/gatsby#2669

Open

[Snyk] Fix for 1 vulnerabilities ajesse11x/gatsby#2670

Open

[Snyk] Fix for 1 vulnerabilities ajesse11x/gatsby#2671

Open

This was referenced Apr 15, 2024

[Snyk] Fix for 1 vulnerabilities MaxMood96/gatsby#1505

Open

[Snyk] Fix for 1 vulnerabilities MaxMood96/gatsby#1506

Open

[Snyk] Fix for 1 vulnerabilities MaxMood96/gatsby#1507

Open

[Snyk] Fix for 1 vulnerabilities MaxMood96/gatsby#1508

Open

impactyogi mentioned this pull request Apr 15, 2024

[Snyk] Fix for 1 vulnerabilities signal-noise/gatsby#1190

Open

MaxMood96 mentioned this pull request Apr 15, 2024

[Snyk] Fix for 1 vulnerabilities MaxMood96/gatsby#1509

Open

This was referenced Apr 15, 2024

[Snyk] Fix for 1 vulnerabilities Xtuden-com/gatsby#5679

Open

[Snyk] Fix for 1 vulnerabilities Xtuden-com/gatsby#5682

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(gatsby-source-drupal): cache backlink records #33444

fix(gatsby-source-drupal): cache backlink records #33444

KyleAMathews commented Oct 5, 2021 •

edited

Loading

wardpeet Oct 25, 2021

KyleAMathews Oct 25, 2021

KyleAMathews Oct 25, 2021

fix(gatsby-source-drupal): cache backlink records #33444

fix(gatsby-source-drupal): cache backlink records #33444

Conversation

KyleAMathews commented Oct 5, 2021 • edited Loading

wardpeet Oct 25, 2021

Choose a reason for hiding this comment

KyleAMathews Oct 25, 2021

Choose a reason for hiding this comment

KyleAMathews Oct 25, 2021

Choose a reason for hiding this comment

KyleAMathews commented Oct 5, 2021 •

edited

Loading