feat: add markdown rendering to content layer

[ascorbic]

Adds support for rendering content. Initially just supports markdown.

This works by rendering the markdown before adding it to the store. A ContentEntryType gets a new optional field getRenderFunction, which returns a reusable render function that accepts an entry object and returns an object with html and metadata fields. The data store set method now accepts rendered as an optional field, which stores these with the rest of the entry.

Users can access the HTML and metadata directly on entry.rendered. Calling entry.render() will return a Content component like now, but in this case the markdown is already rendered to HTML, so its only transform at that point is to replace image references.

To make it easier to optimise rendering, the data store gains a digest field. This is considered an opaque value, but if passed it will be stored and returned in the entry, and if it matches an existing entry in the store then it will skip setting it in the DB. The digest is also used to skip rendering markdown if the text is unchanged.

I have added benchmarks to compare rendering large numbers of markdown docs using content collections v1, and content layer.

This PR includes all of the changes from #11469, so those who reviewed that will see some familiar parts. I asked for that one to be reviewed first because it had some changes to the behaviour of this one.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 76e6a42

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[bholmesdev]

!preview contentlayer

[github-actions]

Snapshots have been released for the following packages:

• astro@experimental--contentlayer

Publish Log

🦋  warn ===============================IMPORTANT!===============================
🦋  warn Packages will be released under the experimental--contentlayer tag
🦋  warn ----------------------------------------------------------------------
🦋  info npm info astro
🦋  info npm info @astrojs/prism
🦋  info npm info @astrojs/rss
🦋  info npm info create-astro
🦋  info npm info @astrojs/db
🦋  info npm info @astrojs/alpinejs
🦋  info npm info @astrojs/lit
🦋  info npm info @astrojs/markdoc
🦋  info npm info @astrojs/mdx
🦋  info npm info @astrojs/node
🦋  info npm info @astrojs/partytown
🦋  info npm info @astrojs/preact
🦋  info npm info @astrojs/react
🦋  info npm info @astrojs/sitemap
🦋  info npm info @astrojs/solid-js
🦋  info npm info @astrojs/svelte
🦋  info npm info @astrojs/tailwind
🦋  info npm info @astrojs/vercel
🦋  info npm info @astrojs/vue
🦋  info npm info @astrojs/web-vitals
🦋  info npm info @astrojs/internal-helpers
🦋  info npm info @astrojs/markdown-remark
🦋  info npm info @astrojs/studio
🦋  info npm info @astrojs/telemetry
🦋  info npm info @astrojs/underscore-redirects
🦋  info npm info @astrojs/upgrade
🦋  info astro is being published because our local version (0.0.0-contentlayer-20240715192053) has not been published on npm
🦋  warn @astrojs/prism is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/rss is not being published because version 4.0.7 is already published on npm
🦋  warn create-astro is not being published because version 4.8.0 is already published on npm
🦋  warn @astrojs/db is not being published because version 0.11.7 is already published on npm
🦋  warn @astrojs/alpinejs is not being published because version 0.4.0 is already published on npm
🦋  warn @astrojs/lit is not being published because version 4.3.0 is already published on npm
🦋  warn @astrojs/markdoc is not being published because version 0.11.1 is already published on npm
🦋  warn @astrojs/mdx is not being published because version 3.1.2 is already published on npm
🦋  warn @astrojs/node is not being published because version 8.3.2 is already published on npm
🦋  warn @astrojs/partytown is not being published because version 2.1.1 is already published on npm
🦋  warn @astrojs/preact is not being published because version 3.5.0 is already published on npm
🦋  warn @astrojs/react is not being published because version 3.6.0 is already published on npm
🦋  warn @astrojs/sitemap is not being published because version 3.1.6 is already published on npm
🦋  warn @astrojs/solid-js is not being published because version 4.4.0 is already published on npm
🦋  warn @astrojs/svelte is not being published because version 5.6.0 is already published on npm
🦋  warn @astrojs/tailwind is not being published because version 5.1.0 is already published on npm
🦋  warn @astrojs/vercel is not being published because version 7.7.2 is already published on npm
🦋  warn @astrojs/vue is not being published because version 4.5.0 is already published on npm
🦋  warn @astrojs/web-vitals is not being published because version 0.2.1 is already published on npm
🦋  warn @astrojs/internal-helpers is not being published because version 0.4.1 is already published on npm
🦋  warn @astrojs/markdown-remark is not being published because version 5.1.1 is already published on npm
🦋  warn @astrojs/studio is not being published because version 0.1.1 is already published on npm
🦋  warn @astrojs/telemetry is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/underscore-redirects is not being published because version 0.3.4 is already published on npm
🦋  warn @astrojs/upgrade is not being published because version 0.3.1 is already published on npm
🦋  info Publishing "astro" at "0.0.0-contentlayer-20240715192053"
🦋  success packages published successfully:
🦋  [email protected]
🦋  Creating git tag...
🦋  New tag:  [email protected]

Build Log

> [email protected] build /home/runner/work/astro/astro
> turbo run build --filter=astro --filter=create-astro --filter="@astrojs/*" --filter="@benchmark/*"

• Packages in scope: @astrojs/alpinejs, @astrojs/cloudflare, @astrojs/db, @astrojs/internal-helpers, @astrojs/lit, @astrojs/markdoc, @astrojs/markdown-remark, @astrojs/mdx, @astrojs/netlify, @astrojs/node, @astrojs/partytown, @astrojs/preact, @astrojs/prism, @astrojs/react, @astrojs/rss, @astrojs/sitemap, @astrojs/solid-js, @astrojs/studio, @astrojs/svelte, @astrojs/tailwind, @astrojs/telemetry, @astrojs/underscore-redirects, @astrojs/upgrade, @astrojs/vercel, @astrojs/vue, @astrojs/web-vitals, @benchmark/timer, astro, create-astro
• Running build in 29 packages
• Remote caching enabled
::group::@astrojs/prism:build
cache miss, executing 88601facc09999bf

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/astro-prism
> astro-scripts build "src/**/*.ts" && tsc -p ./tsconfig.json

::endgroup::
::group::@astrojs/upgrade:build
cache miss, executing f8f66a5298fe1c5f

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/upgrade
> astro-scripts build "src/index.ts" --bundle && tsc

::endgroup::
::group::@astrojs/telemetry:build
cache miss, executing 326dcb16311c5c8f

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/telemetry
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/internal-helpers:build
cache miss, executing 44e186a47257708b

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/internal-helpers
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json


> @astrojs/[email protected] postbuild /home/runner/work/astro/astro/packages/internal-helpers
> astro-scripts copy "src/**/*.js"

::endgroup::
::group::create-astro:build
cache miss, executing c3b6dede768200d5

> [email protected] build /home/runner/work/astro/astro/packages/create-astro
> astro-scripts build "src/index.ts" --bundle && tsc

::endgroup::
::group::@astrojs/markdown-remark:build
cache miss, executing 0e5a326fe43d88e3

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/markdown/remark
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json


> @astrojs/[email protected] postbuild /home/runner/work/astro/astro/packages/markdown/remark
> astro-scripts copy "src/**/*.js"

::endgroup::
::group::astro:build
cache miss, executing d8cd909c466597ff

> [email protected] build /home/runner/work/astro/astro/packages/astro
> pnpm run prebuild && astro-scripts build "src/**/*.{ts,js}" && tsc && pnpm run postbuild


> [email protected] prebuild /home/runner/work/astro/astro/packages/astro
> astro-scripts prebuild --to-string "src/runtime/server/astro-island.ts" "src/runtime/client/{idle,load,media,only,visible}.ts"


> [email protected] postbuild /home/runner/work/astro/astro/packages/astro
> astro-scripts copy "src/**/*.astro" && astro-scripts copy "src/**/*.wasm"

::endgroup::
::group::@astrojs/studio:build
cache miss, executing 8755e2fbd639fa9c

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/studio
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/alpinejs:build
cache miss, executing 66e0ec034314d133

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/alpinejs
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@benchmark/timer:build
cache miss, executing b8a45cab4118abdf

> @benchmark/[email protected] build /home/runner/work/astro/astro/benchmark/packages/timer
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/react:build
cache miss, executing d9f075b8bb448e35

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/react
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/lit:build
cache miss, executing b2c537f255eec4cf

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/lit
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/node:build
cache miss, executing c71958955f4ec128

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/node
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/preact:build
cache miss, executing 6ed38d37927dba77

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/preact
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/markdoc:build
cache miss, executing 5891072d8efccf6f

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/markdoc
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vercel:build
cache miss, executing 811e56e7c2baeb4f

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/vercel
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vue:build
cache miss, executing e3836fd8e10ef8b6

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/vue
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/rss:build
cache miss, executing a680841959e0d812

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/astro-rss
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/partytown:build
cache miss, executing 194957ccfb4e48b6

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/partytown
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/underscore-redirects:build
cache miss, executing aa2423ce739319a3

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json


> @astrojs/[email protected] postbuild /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts copy "src/**/*.js"

::endgroup::
::group::@astrojs/tailwind:build
cache miss, executing 610eab02c7893d67

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/tailwind
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/mdx:build
cache miss, executing f6d5e6ce65b7b216

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/mdx
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/svelte:build
cache miss, executing b2eb693acadbaab8

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/svelte
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/solid-js:build
cache miss, executing 9b4ec91c2224512f

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/solid
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/sitemap:build
cache miss, executing 6871e6b6e7e50bea

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/sitemap
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/db:build
cache miss, executing ca5497e364b36983

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/db
> astro-scripts build "src/**/*.ts" && tsc && pnpm types:virtual


> @astrojs/[email protected] types:virtual /home/runner/work/astro/astro/packages/db
> tsc -p ./tsconfig.virtual.json

::endgroup::
::group::@astrojs/web-vitals:build
cache miss, executing 6e3ee3c5ff34ecb6

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/web-vitals
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::

 Tasks:    27 successful, 27 total
Cached:    0 cached, 27 total
  Time:    58.784s 

[ematipico]

I am not sure I understand the implications of this choice. Can we add some tech documentation somewhere that explains when to use one and when to use the other?

[ematipico]

The value of html can be undefined. Is it possible to have this value? If not, should we throw and error? If yes, should we optimise it? (it feels weird to call createComponent, serverRender and unescapeHTML on something that doesn't need it)

[ascorbic]

What would be the best way to return a component that renders nothing?

[ematipico]

It's possible to shortcircuit it by returning an object like this:

const ComponentInstance = {
	default() {
		return new Response(null, {
			status: 200,
		});
	},
}

However, here we are inside the Content component, so I need to think about it more. Consider this as an optimisation we can do later

[ascorbic]

Thanks

[ematipico]

I am a bit confused on how the PRs have been handled. I was under the impression that this PR would have been merged before the reference PR and the assets PR.

Anyway, looks good to me. I wonder if we could avoid the console.warn and use our logger

[ascorbic]

@ematipico While normally I'd merge the parent branch first, in this case it had been open for quite a while with a lot of work in the child branches since. I needed the assets one merging first because it included some changes to the logic in here that would mean that otherwise you'd be reviewing outdated code.