Skip to content

Commit

Permalink
perf(gatsby): drop severe scaling regression caused by analytics
Browse files Browse the repository at this point in the history
Regression introduced in #22851

The problem seems to be that these calls to `v8.serialize` trigger the gc to start a full hold-the-world mark-and-sweep step sooner. In a benchmark of 150k files, the step would trigger almost always after between 100k and 110k queries had run, and it would pause the process for 60+ seconds.

Example benchmark results from before and after that PR:

```
info bootstrap finished - 86.758 s
success Building production JavaScript and CSS bundles - 9.404s
success run queries - 205.676s - 150002/150002 729.31/s
success Building static HTML for pages - 142.800s - 150002/150002 1050.44/s
info Done building in 451.33 sec
```

```
info bootstrap finished - 85.933 s
success Building production JavaScript and CSS bundles - 8.335s
success run queries - 84.795s - 150002/150002 1769.00/s
success Building static HTML for pages - 141.000s - 150002/150002 1063.84/s
info Done building in 320.158 sec
```

This is very consistent behavior. We looked at the change and agreed that the best was to just drop this measurement since it was for the sake of analytics and a non-vital metric to record. We'd rather have the perf than the metric.

Numbers for the fix, same benchmark, first on current master and then on this PR:

```
info bootstrap finished - 79.788s
success Building production JavaScript and CSS bundles - 9.635s
success run queries - 201.542s - 150002/150002 744.27/s
success Building static HTML for pages - 141.535s - 150002/150002 1059.82/s
info Done building in 440.766 sec
```

```
info bootstrap finished - 80.751s
success Building production JavaScript and CSS bundles - 9.570s
success run queries - 87.162s - 150002/150002 1720.95/s
success Building static HTML for pages - 142.609s - 150002/150002 1051.84/s
info Done building in 319.151 sec
```

nice!
  • Loading branch information
pvdz committed Jun 2, 2020
1 parent 565895b commit 5d109ad
Showing 1 changed file with 12 additions and 16 deletions.
28 changes: 12 additions & 16 deletions packages/gatsby/src/query/graphql-runner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import {
GraphQLSchema,
Source,
GraphQLError,
ExecutionResult,
ExecutionResult
} from "graphql"
import { debounce } from "lodash"
import * as nodeStore from "../db/nodes"
Expand Down Expand Up @@ -42,7 +42,7 @@ export class GraphQLRunner {
protected store: Store<IGatsbyState>,
{
collectStats,
graphqlTracing,
graphqlTracing
}: {
collectStats?: boolean
graphqlTracing?: boolean
Expand All @@ -54,7 +54,7 @@ export class GraphQLRunner {
nodeStore,
schema,
schemaComposer: schemaCustomization.composer,
createPageDependency,
createPageDependency
})
this.schema = schema
this.parseCache = new Map()
Expand All @@ -75,7 +75,7 @@ export class GraphQLRunner {
totalNonSingleFilters: 0,
comparatorsUsed: new Map(),
uniqueFilterPaths: new Set(),
uniqueSorts: new Set(),
uniqueSorts: new Set()
}
} else {
this.stats = null
Expand Down Expand Up @@ -128,7 +128,7 @@ export class GraphQLRunner {
totalNonSingleFilters: this.stats.totalNonSingleFilters,
comparatorsUsed: comparatorsUsedObj,
uniqueFilterPaths: this.stats.uniqueFilterPaths.size,
uniqueSorts: this.stats.uniqueSorts.size,
uniqueSorts: this.stats.uniqueSorts.size
}
} else {
return null
Expand All @@ -140,7 +140,7 @@ export class GraphQLRunner {
context: Record<string, unknown>,
{
parentSpan,
queryName,
queryName
}: { parentSpan: Span | undefined; queryName: string }
): Promise<ExecutionResult> {
const { schema, schemaCustomization } = this.store.getState()
Expand All @@ -156,17 +156,13 @@ export class GraphQLRunner {
if (typeof statsQuery !== `string`) {
statsQuery = statsQuery.body
}
this.stats.uniqueOperations.add(

this.stats.uniqueQueries.add(
crypto
.createHash(`sha1`)
.update(statsQuery)
.update(v8.serialize(context))
.digest(`hex`)
)

this.stats.uniqueQueries.add(
crypto.createHash(`sha1`).update(statsQuery).digest(`hex`)
)
}

const document = this.parse(query)
Expand All @@ -177,8 +173,8 @@ export class GraphQLRunner {
tracer = new GraphQLSpanTracer(`GraphQL Query`, {
parentSpan,
tags: {
queryName: queryName,
},
queryName: queryName
}
})

tracer.start()
Expand All @@ -199,9 +195,9 @@ export class GraphQLRunner {
customContext: schemaCustomization.context,
nodeModel: this.nodeModel,
stats: this.stats,
tracer,
tracer
}),
variableValues: context,
variableValues: context
})

// Queries are usually executed in batch. But after the batch is finished
Expand Down

0 comments on commit 5d109ad

Please sign in to comment.