How webpack cache works

[jerrykingxyz]

Summary

Explain how webpack use cache to improve build speed, and how to implement cache
NOTE: This page is written base on webpack v5.74.0

Glossary

Cache

• Global
  Global cache is a core part of webpack, it exposes basic storage api to hide low-level complex storage logic. all compiler share global cache.

• Local
  Local cache is created by functional class, it is a local variable and is stored in memory.

Snapshot

Snapshot is lifetime information label. webpack checks the difference between before and after snapshots to determine whether the cache is valid. We can config policy as timestamps, content hash or timestamps + content hash.
PS. Snapshot not storage the cached data, it only have lifetime info. (Cache Data = Origin Data + Snapshot)

Unsafe Cache

Webpack provides resolve.unsafeCache and module.unsafeCache to configure cache strength.
PS. resolve.unsafeCache used in enhanced-resolve, module.unsafeCache used in processDependency.

Guide-level explanation

Configuration

We can use the cache field to configure cache in webpack, see this documentation for more subfields

We can use the snapshot field to configure snapshot strategy in webpack, see this documentation for more subfields

Reference-level explanation

Global Cache

Webpack define a basic cache module that provides these hooks to unify the behavior of different caches.

• get(identifier, etag, gotHandlers: Array<(result, callback) => void>) -> result
  Get data from cache. the gotHandlers is a post-process function array, which handles the final result. Memory cache use gotHandlers to store the final result.

• store(identifier, etag, data)
  Save data to cache

• storeBuildDependencies(dependencies)
  Save build dependencies to cache

• beginIdle()
  Run hooks when the build task begin idle

• endIdle()
  Run hooks when the build task end idle

• shutdown()
  Run hooks when need clear the cache

All of those hooks are provide a same name function for easy calling. Webpack has three preset cache:

• MemoryCache
  Use memory to cache intermediate results for build.

const cache = new Map<string, { etag: Etag | null, data: any }>();

• MemoryWithGcCache
  Use memory to cache intermediate results for build, but clear cache when out of lifespan (lifespan config by cache.maxGenerations)

const cache = new Map<string, { etag: Etag | null, data: any }>();
const oldCache = new Map<string, { etag: Etag | null, data: any }>();

// when compiler after done will clear oldCache and copy cache to oldCache

• IdleFileCache
  Use file system to cache intermediate results for build. this cache will generate a queue of tasks that write cache to fs when build is idle.

const pendingIdleTasks = new Map<string, () => Promise>();

// store
pendingIdleTasks.set(identifier, () => {
  // the strategy provide method to write data to file system
  strategy.store(identifier, etag, data)
})

Snapshot

The snapshot is implemented in FileSystemInfo.js.

class Snapshot {
  _flags: number
  startTime?: number
  fileTimestamps?: Map<string, FileSystemInfoEntry | null>
  fileHashes?: Map<string, string | null>
  fileTshs?: Map<string, TimestampAndHash | string | null>
  ...
}

// create
function createSnapshot(startTime, files, directories, missing, options, callback) {}

// valid
checkSnapshotValid(snapshot, callback) {}

Use cache

• Resolve module
  Webpack read module data and snapshot from cache and skips module resolve by checking if the snapshot is valid.

• Build module
  Webpack use module.needBuild to check whether current need build. webpack will check module cacheable & snapshot in the module.needBuild.

• Process dependency
  Use module.unsafe to config module dependencies need to skip resolve.

• Code generate
  Skip code generation by filtering cached code by module hash and identifier.

• Chunk assets
  Use cached chunk assets with asset identifier.

• Emit assets
  Use cached assets with asset identifier.