Content Collections (#5291)

* feat: port content plugin as-is

* feat: add `fetchContent` to `.astro`

* refactor: move more logic to content-internals

* refactor: move internals -> content dir

* feat: nested dir support

* feat: add with-content playground

* edit: update README

* fix: serialize route pattern for Netlify edge

Co-authored-by: Jackie Macharia <jackiewmacharia>

* wip: add ?astro-asset-ssr flag

* wip: exclude astro-ssr-asset styles from build

* revert: SSR asset flag babel changes

* feat: only load CSS when mdx is rendered!

* fix(vercel): Include all files inside `dist/` instead of only `entry.mjs` (#5175)

* chore: only-allow pnpm (#5131)

* [MDX] Support `recmaPlugins` config (#5146)

* feat: support recma plugins

* chore: add `test:match` to MDX

* chore: changeset

* docs: add recmaPlugins to README

* removes default content-visibility styles from image components (#5180)

* wip: check renderContent works

* nit: injectDelayedAssetPlugin doesn't need enforce

* feat: render content with generated entries

* feat: renderContent style injection POC

* wip: scrappy renderContent binding

* deps: parse-imports

* feat: only bind renderContent when imported

* feat: new-and-improved render-content

* fix: update example to use render-content

* refactor: extract normalizeFilename helper

* feat: move renderContent to `.astro`

* feat: add `getStaticPaths` demo 👀

* chore: remove console log

* refactor: internals -> internal

* fix: style bleed on dynamic routes

* chore: remove console log

* refactor: move rendercontentmap inside same file

* fix: style bleed in dev!

* feat: clean up example

* refactor: curry generateContent

* nit: add datetime demo

* refactor: rename and move content plugins

* feat: granular content updates in development

* chore: console.log

* refactor: move write logic, fix edge cases

* fix: use z.any() when ~schema is missing

* refactor: clean up logging

* fix: add delayed asset flag to render map

* fix: use new server loader

* chore: import type

* fix: split delayed assets to separate chunks

* chore: bad merge changesets

* feat: rework content map to Vite globs

* fix: formatted errors

* fix: get `renderContent` working with new APIs

* refactor: rename to getCollection, getEntry, renderEntry

* feat: streamline type alias!

* fix: check schema import is an import

* refactor: change ~schema -> index with defineCollection

* feat: auto apply env.d.ts reference

* feat: expose `headings` on `renderEntry`

* refactor: expose zod from astro/zod

* feat: add Collection util type

* refactor: generated types in src/content/

* edit: remove env.d.ts change from with-content

* feat: collectionToPaths utility

* refactor: migrate `examples/blog`

* refactor: simplify [...slug]

* fix: invalid date

* fix: renderEntry

* feat: migrate examples/docs to content!

* refactor: union -> enum

* fix: remove duplicate header

* chore: changeset

* Revert "fix: serialize route pattern for Netlify edge"

This reverts commit ac46a0d.

* fix: lint

* fix: include zod in package files

* chore: push type gen for astro check

* feat: move to single defineCollections file

* feat: move examples to new collections model

* chore: type check valid collection names

* fix: ignore collections path in err logs

* refactor: content/index -> content/config

* refactor: defineCollections -> defineCollection

* fix: update docs to defineCollection

* chore: update with-content to new `astro:content` setup

* fix: renderEntry styles

* fix: bad Collection type

* fix: blog types generated

* feat: expose injectedFrontmatter on renderEntry

* chore: `import type`

* chore: add MDX to changeset

* fix: template package file

* fix: unneeded log on content config change

* chore: add `injectedFrontmatter` to generated types

* fix: escape import.meta.env in `body`

* fix: escape import.meta.env in frontmatter

* chore: unused import

* fix: use URL for config pathname

* feat: allow zod async transforms

* deps: devalue

* feat: custom slugs + better type checking

* fix: update types on config added or removed

* chore: add error msg for MDX preview release

* refactor: restore old normalizeFilename

* fix: add loadContentConfig error classes back

* Add initial tests for renderEntry

* add fixture tests

* Use head propagation APIs in renderEntry

* chore: new type gen

* fix: sort content keys before writing

* fix: avoid writing '.' collection with warning

* chore: docs types generated

* fix: generated types only once in build

* fix: use relative path to config for git

* chore: update generated types

* fix: glob pattern on valid exts

* feat: warn on possible collection typos

* chore: better error on bad lazy import

* fix: clear gray-matter cache for rawData

* fix: wait for config file while generating

* test: get collection and get entry basic

* test: multiple collections and entries

* test: add publishedDate test on entry

* tests: getStaticPaths integration

* test: ssr endpoint integration

* chore: consistent test names

* refactor: simplify generates... test

* chore: tests lockfile

* feat: add `experimental` flag

* chore: revert blog and docs examples

* chore: add experimental flag to with-content

* refactor: reuse contentFileExts

* refactor: make contentCollectionCss required

* chore: remove parse-imports dependency

* chore: remove unresolved idea redundancy

* chore: update changeset to minor v

* chore: clean up with-content pass 1

* refactor: de-monolith vite-plugin-content

* chore: remove maximum jank warning (it's acceptable jank)

* refactor: extract types-generator to separate file

* refactor: onEvent -> handleEvent

* edit: change content log to "types generated"

* fix: more experimental checks in prod build

* refactor: 2 jsToTreeNode calls -> 1

* feat: change with-content example to blog example

* fix: Content component type

* Fix use of renderEntry within a slot

* fix: warn for non-underscore _ files, refine logs

* chore: Content type in tests

* fix: types no updating with config changes

* fix: remove `result: SSRResult` type

* chore: entry var shadowing

* fix: bad experimental flag call

* wip: skip render entry tests

* fix: path resolution for .relative on windows

* chore: remove dead code

* wip: log entryPath into

* fix: use URL for entry

* refactor: getEntryInfo normalization

* refactor: move debounce to queueEvent only

* fix: resolve loader on config error

* fix: pathToFileURL -> new URL

* fix: dangerous pathname comparison

* refactor: move rawEvent -> event mapping to queueEvent

* fix: bad type

* wip: remove mkdir call

* wip: log content paths

* fix: more bad paths

* wip: more logs

* fix: correct glob path

* fix: normalize virtual mod path

* fix: content server file paths

* chore: remove render entry test skip

* wip: skip renderEntry tests (windows issues?)

* chore: remove console log

* feat: renderEntry -> render

* chore: add `--experimental-content-collections`

* refactor: only skip renderEntry tests on windows

* refactor: index.ts barrel export

* fix: cli type error

* fix: proper test skip on windows

* refactor: remove collectionToPaths :(

* refactor: use AstroError for frontmatter error

* nit: its schema, .*

* fix: bad isWindows import

* fix: await mod -> mod = await

Co-authored-by: Juan Martín Seery <[email protected]>
Co-authored-by: 花果山大圣 <[email protected]>
Co-authored-by: Tony Sullivan <[email protected]>
Co-authored-by: Matthew Phillips <[email protected]>
Co-authored-by: bholmesdev <[email protected]>

.changeset/heavy-plums-end.md

@@ -0,0 +1,9 @@
+---
+'astro': minor
+'@astrojs/mdx': minor
+---
+
+Introduce Content Collections experimental API
+- Organize your Markdown and MDX content into easy-to-manage collections.
+- Add type safety to your frontmatter with schemas.
+- Generate landing pages, static routes, and SSR endpoints from your content using the collection query APIs.

examples/with-content/.gitignore

@@ -0,0 +1,19 @@
+# build output
+dist/
+
+# dependencies
+node_modules/
+
+# logs
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+
+
+# environment variables
+.env
+.env.production
+
+# macOS-specific files
+.DS_Store

examples/with-content/.vscode/extensions.json

@@ -0,0 +1,4 @@
+{
+  "recommendations": ["astro-build.astro-vscode"],
+  "unwantedRecommendations": []
+}

examples/with-content/.vscode/launch.json

@@ -0,0 +1,11 @@
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "command": "./node_modules/.bin/astro dev",
+      "name": "Development server",
+      "request": "launch",
+      "type": "node-terminal"
+    }
+  ]
+}

examples/with-content/README.md

@@ -0,0 +1,52 @@
+# Astro Content Collections (Experimental)
+
+This demos our Blog Starter using **[the experimental Content Collections API.](https://docs.astro.build/en/guides/content-collections)**
+
+[![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/withastro/astro/tree/latest/examples/with-content)
+[![Open with CodeSandbox](https://assets.codesandbox.io/github/button-edit-lime.svg)](https://codesandbox.io/s/github/withastro/astro/tree/latest/examples/with-content)
+
+> 🧑‍🚀 **Seasoned astronaut?** Delete this file. Have fun!
+
+## 🚀 Project Structure
+
+Inside of your Astro project, you'll see the following folders and files:
+
+```
+├── public/
+├── src/
+│   └── content/
+│       └── blog/
+│           ├── first.md
+│           └── second.md
+│       └── config.ts
+│   ├── components/
+│   ├── layouts/
+│   └── pages/
+├── astro.config.mjs
+├── README.md
+├── package.json
+└── tsconfig.json
+```
+
+`src/content/` contains "collections" of related Markdown and MDX documents. Astro will generate a `getCollection` function to retrieve posts from `src/content/`, and type-check your frontmatter using an optional schema (see `src/content/config.ts`).
+
+## 🧞 Commands
+
+All commands are run from the root of the project, from a terminal:
+
+| Command                | Action                                           |
+| :--------------------- | :----------------------------------------------- |
+| `npm install`          | Installs dependencies                            |
+| `npm run dev`          | Starts local dev server at `localhost:3000`      |
+| `npm run build`        | Build your production site to `./dist/`          |
+| `npm run preview`      | Preview your build locally, before deploying     |
+| `npm run astro ...`    | Run CLI commands like `astro add`, `astro check` |
+| `npm run astro --help` | Get help using the Astro CLI                     |
+
+## 👀 Want to learn more?
+
+Check out [our documentation](https://docs.astro.build) or jump into our [Discord server](https://astro.build/chat).
+
+## Credit
+
+This theme is based off of the lovely [Bear Blog](https://github.com/HermanMartinus/bearblog/).

examples/with-content/astro.config.mjs

@@ -0,0 +1,13 @@
+import { defineConfig } from 'astro/config';
+import mdx from '@astrojs/mdx';
+
+import sitemap from '@astrojs/sitemap';
+
+// https://astro.build/config
+export default defineConfig({
+	site: 'https://example.com',
+	integrations: [mdx(), sitemap()],
+	experimental: {
+		contentCollections: true,
+	},
+});

examples/with-content/package.json

@@ -0,0 +1,18 @@
+{
+  "name": "@example/with-content",
+  "type": "module",
+  "version": "0.0.1",
+  "private": true,
+  "scripts": {
+    "dev": "astro dev",
+    "start": "astro dev",
+    "build": "astro build",
+    "preview": "astro preview",
+    "astro": "astro"
+  },
+  "dependencies": {
+    "astro": "^1.6.15",
+    "@astrojs/mdx": "^0.12.2",
+    "@astrojs/sitemap": "^1.0.0"
+  }
+}

examples/with-content/sandbox.config.json

@@ -0,0 +1,11 @@
+{
+  "infiniteLoopProtection": true,
+  "hardReloadOnChange": false,
+  "view": "browser",
+  "template": "node",
+  "container": {
+    "port": 3000,
+    "startScript": "start",
+    "node": "14"
+  }
+}

examples/with-content/src/components/BaseHead.astro

@@ -0,0 +1,38 @@
+---
+// Import the global.css file here so that it is included on
+// all pages through the use of the <BaseHead /> component.
+import '../styles/global.css';
+
+export interface Props {
+	title: string;
+	description: string;
+	image?: string;
+}
+
+const { title, description, image = '/placeholder-social.jpg' } = Astro.props;
+---
+
+<!-- Global Metadata -->
+<meta charset="utf-8" />
+<meta name="viewport" content="width=device-width,initial-scale=1" />
+<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
+<meta name="generator" content={Astro.generator} />
+
+<!-- Primary Meta Tags -->
+<title>{title}</title>
+<meta name="title" content={title} />
+<meta name="description" content={description} />
+
+<!-- Open Graph / Facebook -->
+<meta property="og:type" content="website" />
+<meta property="og:url" content={Astro.url} />
+<meta property="og:title" content={title} />
+<meta property="og:description" content={description} />
+<meta property="og:image" content={new URL(image, Astro.url)} />
+
+<!-- Twitter -->
+<meta property="twitter:card" content="summary_large_image" />
+<meta property="twitter:url" content={Astro.url} />
+<meta property="twitter:title" content={title} />
+<meta property="twitter:description" content={description} />
+<meta property="twitter:image" content={new URL(image, Astro.url)} />

examples/with-content/src/components/Footer.astro

@@ -0,0 +1,13 @@
+---
+const today = new Date();
+---
+
+<footer>
+	&copy; {today.getFullYear()} YOUR NAME HERE. All rights reserved.
+</footer>
+<style>
+	footer {
+		padding: 25px;
+		text-align: center;
+	}
+</style>

examples/with-content/src/components/Header.astro

@@ -0,0 +1,25 @@
+---
+import HeaderLink from './HeaderLink.astro';
+import { SITE_TITLE } from '../consts';
+---
+
+<header>
+	<h2>
+		{SITE_TITLE}
+	</h2>
+	<nav>
+		<HeaderLink href="/">Home</HeaderLink>
+		<HeaderLink href="/blog">Blog</HeaderLink>
+		<HeaderLink href="/about">About</HeaderLink>
+		<HeaderLink href="https://twitter.com/astrodotbuild" target="_blank">Twitter</HeaderLink>
+		<HeaderLink href="https://github.com/withastro/astro" target="_blank">GitHub</HeaderLink>
+	</nav>
+</header>
+<style>
+	header {
+		margin: 0em 0 2em;
+	}
+	h2 {
+		margin: 0.5em 0;
+	}
+</style>

examples/with-content/src/components/HeaderLink.astro

@@ -0,0 +1,22 @@
+---
+export interface Props extends astroHTML.JSX.AnchorHTMLAttributes {}
+
+const { href, class: className, ...props } = Astro.props;
+
+const { pathname } = Astro.url;
+const isActive = href === pathname || href === pathname.replace(/\/$/, '');
+---
+
+<a href={href} class:list={[className, { active: isActive }]} {...props}>
+	<slot />
+</a>
+<style>
+	a {
+		display: inline-block;
+		text-decoration: none;
+	}
+	a.active {
+		font-weight: bolder;
+		text-decoration: underline;
+	}
+</style>

examples/with-content/src/consts.ts

@@ -0,0 +1,5 @@
+// Place any global data in this file.
+// You can import this data from anywhere in your site by using the `import` keyword.
+
+export const SITE_TITLE = 'My personal website.';
+export const SITE_DESCRIPTION = 'Welcome to my website!';

examples/with-content/src/content/blog/first-post.md

@@ -0,0 +1,16 @@
+---
+title: "First post"
+description: "Lorem ipsum dolor sit amet"
+pubDate: "Jul 08 2022"
+heroImage: "/placeholder-hero.jpg"
+---
+
+Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Vitae ultricies leo integer malesuada nunc vel risus commodo viverra. Adipiscing enim eu turpis egestas pretium. Euismod elementum nisi quis eleifend quam adipiscing. In hac habitasse platea dictumst vestibulum. Sagittis purus sit amet volutpat. Netus et malesuada fames ac turpis egestas. Eget magna fermentum iaculis eu non diam phasellus vestibulum lorem. Varius sit amet mattis vulputate enim. Habitasse platea dictumst quisque sagittis. Integer quis auctor elit sed vulputate mi. Dictumst quisque sagittis purus sit amet.
+
+Morbi tristique senectus et netus. Id semper risus in hendrerit gravida rutrum quisque non tellus. Habitasse platea dictumst quisque sagittis purus sit amet. Tellus molestie nunc non blandit massa. Cursus vitae congue mauris rhoncus. Accumsan tortor posuere ac ut. Fringilla urna porttitor rhoncus dolor. Elit ullamcorper dignissim cras tincidunt lobortis. In cursus turpis massa tincidunt dui ut ornare lectus. Integer feugiat scelerisque varius morbi enim nunc. Bibendum neque egestas congue quisque egestas diam. Cras ornare arcu dui vivamus arcu felis bibendum. Dignissim suspendisse in est ante in nibh mauris. Sed tempus urna et pharetra pharetra massa massa ultricies mi.
+
+Mollis nunc sed id semper risus in. Convallis a cras semper auctor neque. Diam sit amet nisl suscipit. Lacus viverra vitae congue eu consequat ac felis donec. Egestas integer eget aliquet nibh praesent tristique magna sit amet. Eget magna fermentum iaculis eu non diam. In vitae turpis massa sed elementum. Tristique et egestas quis ipsum suspendisse ultrices. Eget lorem dolor sed viverra ipsum. Vel turpis nunc eget lorem dolor sed viverra. Posuere ac ut consequat semper viverra nam. Laoreet suspendisse interdum consectetur libero id faucibus. Diam phasellus vestibulum lorem sed risus ultricies tristique. Rhoncus dolor purus non enim praesent elementum facilisis. Ultrices tincidunt arcu non sodales neque. Tempus egestas sed sed risus pretium quam vulputate. Viverra suspendisse potenti nullam ac tortor vitae purus faucibus ornare. Fringilla urna porttitor rhoncus dolor purus non. Amet dictum sit amet justo donec enim.
+
+Mattis ullamcorper velit sed ullamcorper morbi tincidunt. Tortor posuere ac ut consequat semper viverra. Tellus mauris a diam maecenas sed enim ut sem viverra. Venenatis urna cursus eget nunc scelerisque viverra mauris in. Arcu ac tortor dignissim convallis aenean et tortor at. Curabitur gravida arcu ac tortor dignissim convallis aenean et tortor. Egestas tellus rutrum tellus pellentesque eu. Fusce ut placerat orci nulla pellentesque dignissim enim sit amet. Ut enim blandit volutpat maecenas volutpat blandit aliquam etiam. Id donec ultrices tincidunt arcu. Id cursus metus aliquam eleifend mi.
+
+Tempus quam pellentesque nec nam aliquam sem. Risus at ultrices mi tempus imperdiet. Id porta nibh venenatis cras sed felis eget velit. Ipsum a arcu cursus vitae. Facilisis magna etiam tempor orci eu lobortis elementum. Tincidunt dui ut ornare lectus sit. Quisque non tellus orci ac. Blandit libero volutpat sed cras. Nec tincidunt praesent semper feugiat nibh sed pulvinar proin gravida. Egestas integer eget aliquet nibh praesent tristique magna.

examples/with-content/src/content/blog/markdown-style-guide.md

@@ -0,0 +1,102 @@
+---
+title: "Markdown Style Guide"
+description: "Here is a sample of some basic Markdown syntax that can be used when writing Markdown content in Astro."
+pubDate: "Jul 01 2022"
+heroImage: "/placeholder-hero.jpg"
+---
+
+Here is a sample of some basic Markdown syntax that can be used when writing Markdown content in Astro.
+
+## Headings
+
+The following HTML `<h1>`—`<h6>` elements represent six levels of section headings. `<h1>` is the highest section level while `<h6>` is the lowest.
+
+# H1
+## H2
+### H3
+#### H4
+##### H5
+###### H6
+
+## Paragraph
+
+Xerum, quo qui aut unt expliquam qui dolut labo. Aque venitatiusda cum, voluptionse latur sitiae dolessi aut parist aut dollo enim qui voluptate ma dolestendit peritin re plis aut quas inctum laceat est volestemque commosa as cus endigna tectur, offic to cor sequas etum rerum idem sintibus eiur? Quianimin porecus evelectur, cum que nis nust voloribus ratem aut omnimi, sitatur? Quiatem. Nam, omnis sum am facea corem alique molestrunt et eos evelece arcillit ut aut eos eos nus, sin conecerem erum fuga. Ri oditatquam, ad quibus unda veliamenimin cusam et facea ipsamus es exerum sitate dolores editium rerore eost, temped molorro ratiae volorro te reribus dolorer sperchicium faceata tiustia prat.
+
+Itatur? Quiatae cullecum rem ent aut odis in re eossequodi nonsequ idebis ne sapicia is sinveli squiatum, core et que aut hariosam ex eat.
+
+## Images
+
+![This is a placeholder image description](/placeholder-social.jpg)
+
+## Blockquotes
+
+The blockquote element represents content that is quoted from another source, optionally with a citation which must be within a `footer` or `cite` element, and optionally with in-line changes such as annotations and abbreviations.
+
+#### Blockquote without attribution
+
+> Tiam, ad mint andaepu dandae nostion secatur sequo quae.  
+> **Note** that you can use *Markdown syntax* within a blockquote.
+
+#### Blockquote with attribution
+
+> Don't communicate by sharing memory, share memory by communicating.<br>
+> — <cite>Rob Pike[^1]</cite>
+
+[^1]: The above quote is excerpted from Rob Pike's [talk](https://www.youtube.com/watch?v=PAAkCSZUG1c) during Gopherfest, November 18, 2015.
+
+## Tables
+
+| Italics   | Bold     | Code   |
+| --------  | -------- | ------ |
+| *italics* | **bold** | `code` |
+
+## Code Blocks
+
+```html
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>Example HTML5 Document</title>
+</head>
+<body>
+  <p>Test</p>
+</body>
+</html>
+```
+
+## List Types
+
+#### Ordered List
+
+1. First item
+2. Second item
+3. Third item
+
+#### Unordered List
+
+* List item
+* Another item
+* And another item
+
+#### Nested list
+
+* Fruit
+  * Apple
+  * Orange
+  * Banana
+* Dairy
+  * Milk
+  * Cheese
+
+## Other Elements — abbr, sub, sup, kbd, mark
+
+<abbr title="Graphics Interchange Format">GIF</abbr> is a bitmap image format.
+
+H<sub>2</sub>O
+
+X<sup>n</sup> + Y<sup>n</sup> = Z<sup>n</sup>
+
+Press <kbd><kbd>CTRL</kbd>+<kbd>ALT</kbd>+<kbd>Delete</kbd></kbd> to end the session.
+
+Most <mark>salamanders</mark> are nocturnal, and hunt for insects, worms, and other small creatures.