feat: add typegen for loaders #11358

ascorbic · 2024-06-27T06:29:35Z

Changes

Adds typegen for loaders. Unlike collections where the schema is defined by the user, these can't be inferred. This is because they can be generated dynamically by the loader during sync. This may happen for example when introspecting a CMS. Because of this, we use zod-to-ts to generate the actual types during sync.

Testing

Docs

changeset-bot · 2024-06-27T06:29:40Z

⚠️ No Changeset found

Latest commit: e98c6d5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

ascorbic · 2024-06-27T06:37:31Z

packages/astro/src/content/loaders.ts

@@ -30,13 +30,13 @@ export interface LoaderContext {
 	): T;
 }

-export interface Loader<S extends ZodSchema = ZodSchema> {
+export interface Loader {


Making this generic messed up inference for top level schemas

packages/astro/tsconfig.json

ematipico · 2024-06-27T08:08:54Z

Before merging this one, can you create a PR for these changes #11334 ?

I know we are going to use that PR for preview releases and the final release, but it would be useful to have a PR where we can actually add comments without littering the original PR, which will be used for docs

ascorbic · 2024-06-27T08:10:19Z

@ematipico sorry, I don't understand. A PR from which branch to which branch?

ematipico · 2024-06-27T08:15:51Z

@ematipico sorry, I don't understand. A PR from which branch to which branch?

Something like this

final PR: main <- contect-collections-v2 (not needed right now, but you can open it anyway as a draft)
then a PR contect-collections-v2 <- content-layer-loader, so we can review and eventually merge it
then this PR will be contect-collections-v2 <- loader-types after we merged the previous PR, before merging it, it will be content-layer-loader <- loader-types

ascorbic · 2024-06-27T08:17:04Z

OK, so the new PR would be a branch that's currently identical to main, and then #11334 would be changed to target that?

ematipico · 2024-06-27T08:19:19Z

Yes exactly

ascorbic · 2024-06-27T08:25:31Z

OK:

added Content Layer #11360 for the base branch.
changed feat: add Content Layer loader #11334 to target that branch
this one is still stacked on feat: add Content Layer loader #11334

florian-lefebvre · 2024-06-27T08:35:23Z

packages/astro/src/content/types-generator.ts

@@ -5,6 +5,7 @@ import glob from 'fast-glob';
 import { bold, cyan } from 'kleur/colors';
 import { type ViteDevServer, normalizePath } from 'vite';
 import { z } from 'zod';
+import { zodToTs, printNode } from 'zod-to-ts';


We'll have to document the limitations of this library, iirc some types can't be generated

* Empty commit * Changeset * feat: add Content Layer loader (#11334) * wip * wip * wip * Update demo * Add meta * wip * Add file loader * Add schema validation * Remove log * Changeset * Format * Lockfile * Fix type * Handle loading for data store JSON * Use rollup util to import JSON * Fix types * Format * Add tests * Changes from review * fix: sync content layer in dev (#11365) * wip * wip * wip * Update demo * Add meta * wip * Add file loader * Add schema validation * Remove log * Changeset * Format * Lockfile * Fix type * Handle loading for data store JSON * Use rollup util to import JSON * Fix types * Format * Add tests * Changes from review * Sync content layer in dev * feat: add typegen for loaders (#11358) * fix: watch for content layer changes (#11371) * fix: watch for content layer changes * Add test * feat: adds simple loader (#11386) * wip * Add simple loader * Fix type guard * Tighten loader schema * Add loader function to type * Reinstall vitest * feat: add glob loader (#11398) * feat: add glob loader * Enable watching and fix paths * Store the full entry object, not just data * Add generateId support * Fix test * Rename loaders to sync * Refacctor imports * Use getEntry * Format * Fix import * Remove type from output * Windows path * Add test for absolute path * Update lockfile * Debugging windows * Allow file URL for base dir * Reset time limit * feat: add markdown rendering to content layer (#11440) * feat: add glob loader * Enable watching and fix paths * Store the full entry object, not just data * Add generateId support * Fix test * Rename loaders to sync * Refacctor imports * Use getEntry * Format * Fix import * Remove type from output * Windows path * Add test for absolute path * Update lockfile * Debugging windows * Allow file URL for base dir * Reset time limit * wip: add markdown rendering to content layer * use cached entries * CLean up types * Instrument more of the build * Add digest helper * Add comments * Make image extraction work * feat: image support for content layer (#11469) * wip * wip * Add image to benchmark * Stub assets if missing * Resolve assets in data * Ignore virtual module * Format * rm log * Handle images when using cached data * Fix CCC * Add a comment * Changes from review * Format * Use relative paths for asset files * Pass all md props to getImage * Ensure dotastro dir exists * Fix tests * Changes from review * Don't use temp array in getcollection * Add error handling * Format * Handle paths that are already relative * Dedupe sync runs * Fix syncing in dev * Changes from review * Windows paths ftw * feat(content-layer): support references in content layer (#11494) * Support references in content layer * Fix utf8 rendering * Warn for invalid entries * Fix test * lol windows paths * Remove assertion * chore: fix content layer types (#11527) * Add experimental_content type * Fix import * Make data store methods generic * fix loader types * Lockfile * Clean content layer with `--force` (#11541) * Clearn content layer with `--force` * Add tests * Document --force flag * Fixes to content layer render types (#11558) * Lockfile * feat: use devalue to serialize content layer data (#11562) * feat: use devalue to serialize content layer data * Fix import * Use devalue stringify * Unused import * Propagate error messages correctly * Support --force flag in sync and dev (#11581) * Support --force flag in sync and dev * Fix test * Separate render function and merge content layer types (#11579) * Separate render function and merge content layer types * Changes from review * fix: clear content layer cache if config has changed (#11591) * fix: clear content layer cache if config has changed * Add test * Watch config * Change from review * fix: skip glob files in content dir (#11622) * fix: skip glob files in content dir * Changes from review * Log pattern * Refactor content layer into shared instance (#11625) * Refactor content layer into shared instance * Clean up when testing * Handle cleanup * fix: support filters in content layer getCollection (#11631) * Throw when using deprecated getEntryByX functions with content layer (#11637) * Updates to content layer types and jsdocs (#11643) * Add hot key to reload content layer (#11626) * Add hot key to reload content layer * Fix filename * Remove cli message * Update example * Change key to "s" * feat: handle simple mdx rendering (#11633) * feat: handle simple mdx rendering * cleanup * feedback * fix regression * remove log * flip condition * update tests * log collections to understand the error * let's try this alternative * try parallel test to understand the issue * chore: use a new fixture to fix tests * rebase and docs * fix regressions * remove old code * address feedback * rename param * log error * rebase * chore: try a different cache dir to solve the error test * fix invalidation of the module when there's no store available * address suggestion * run formatter * update lock file * Lint * Add experimental content layer flag (#11652) * Add experimental content layer flag * Syntax and format * Aside * Format * Reset content config between runs * Update fixture * Update terminology * Lint * wut * Normalize render function return value (#11663) * Add markdoc support to content layer (#11664) * Add markdoc support to content layer * Switch test to cheerio * Update benchmarks * update lock file * Update content layer flag docs (#11682) * Update content layer flag docs * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <[email protected]> * More markdoc * Typo * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <[email protected]> * Update --------- Co-authored-by: Sarah Rainsberger <[email protected]> * Add changeset for content layer experimental release (#11644) * Add changeset for content layer experimental release * Update changeset * Update .changeset/smooth-chicken-wash.md Co-authored-by: Sarah Rainsberger <[email protected]> --------- Co-authored-by: Sarah Rainsberger <[email protected]> * feat: injectTypes (#11551) * feat: make inline config 1st arg * fix: run config done in sync * feat: start working on injectTypes * feat: write files * feat: adapt core features * feat: migrate db to injectTypes * feat: special db handling * feat: update settings instead of workarounds * fix: create dotAstroDir * feat: refactor sync tests * fix: path * fix: paths * chore: add comments * feat: overwrite content file if exists * chore: remove unused db env related code * feat: use dotAstroDir for settings * chore: simplify astro env sync * feat: use dotAstroDir for preferences * feat: handle db in integration api * chore: reorganize * feat: format * feat: add test * Discard changes to examples/basics/astro.config.mjs * Discard changes to examples/basics/package.json * Discard changes to pnpm-lock.yaml * chore: remove test files * feat: update examples dts * fix: dts * chore: changesets * fix: indentation * Apply suggestions from code review Co-authored-by: Chris Swithinbank <[email protected]> * Apply suggestions from code review Co-authored-by: Chris Swithinbank <[email protected]> * chore: format * Update packages/astro/src/integrations/hooks.ts * Update .changeset/mean-horses-kiss.md * feat: remove formatting * feat: handle fs errors * feat: remove astro:db special path handling * feat: add fs error * Update packages/astro/src/content/types-generator.ts * Update .changeset/mean-horses-kiss.md * Update errors-data.ts * Update .changeset/mean-horses-kiss.md Co-authored-by: Sarah Rainsberger <[email protected]> * Update .changeset/mean-horses-kiss.md Co-authored-by: Sarah Rainsberger <[email protected]> --------- Co-authored-by: Chris Swithinbank <[email protected]> Co-authored-by: Sarah Rainsberger <[email protected]> Co-authored-by: Erika <[email protected]> * Add file generation and flag for content intellisense (#11639) * feat: add type to infer input type of collection * refactor: * feat: generate json schema for content too * feat: generate a manifest of all the collections * refactor: unnecessary type * fix: only add content collections to manifest * chore: changeset * fix: generate file URLs * fix: flag it properly * fix: save in lower case * docs: add jsdoc to experimental option * nit: move function out * fix: match vscode flag name * Update packages/astro/src/@types/astro.ts Co-authored-by: Sarah Rainsberger <[email protected]> * Update packages/astro/src/@types/astro.ts Co-authored-by: Sarah Rainsberger <[email protected]> * Update serious-pumas-run.md * test: add tests * Add content layer support * Apply suggestions from code review * fix: test * Update .changeset/serious-pumas-run.md Co-authored-by: Sarah Rainsberger <[email protected]> * Apply suggestions from code review * Remove check for json --------- Co-authored-by: Matt Kane <[email protected]> Co-authored-by: Sarah Rainsberger <[email protected]> * nit: use same filesystem error as injectTypes * fix: code component was missing support for meta string (#11605) * fix: code component was missing support for meta string Fixed #11604 * Create odd-buttons-pay.md * <Code>: add reference link for meta prop * Apply suggestions from code review Co-authored-by: Sarah Rainsberger <[email protected]> * Update .changeset/odd-buttons-pay.md * Update .changeset/odd-buttons-pay.md Co-authored-by: Sarah Rainsberger <[email protected]> --------- Co-authored-by: Erika <[email protected]> Co-authored-by: Sarah Rainsberger <[email protected]> Co-authored-by: Emanuele Stoppa <[email protected]> * Deprecates exporting prerender with dynamic values (#11657) * wip * done i think * Add changeset * Use hook instead * Reorder hooks [skip ci] * Update .changeset/eleven-pens-glow.md Co-authored-by: Sarah Rainsberger <[email protected]> * Fix run * Fix link * Add link Co-authored-by: Sarah Rainsberger <[email protected]> * More accurate migration [skip ci] --------- Co-authored-by: Sarah Rainsberger <[email protected]> Co-authored-by: Sarah Rainsberger <[email protected]> * Use node parseArgs instead of yargs-parser and arg (#11645) * wip * done * Add changeset * Format * Update * Fix houston * Fix test * Fix test * [ci] format * resolve conflict --------- Co-authored-by: Emanuele Stoppa <[email protected]> Co-authored-by: Sarah Rainsberger <[email protected]> Co-authored-by: Florian Lefebvre <[email protected]> Co-authored-by: Chris Swithinbank <[email protected]> Co-authored-by: Erika <[email protected]> Co-authored-by: Julien Cayzac <[email protected]> Co-authored-by: Bjorn Lu <[email protected]> Co-authored-by: Sarah Rainsberger <[email protected]> Co-authored-by: Bjorn Lu <[email protected]>

ascorbic added 21 commits June 24, 2024 17:24

wip

aa1db49

wip

a7639e3

wip

63a23d2

Update demo

624c377

Add meta

d67ee7e

wip

631f3d2

Add file loader

4b7f8f7

Add schema validation

14330f7

Remove log

b3b8dd4

Merge branch 'main' into content-layer-loader

04d09f0

Changeset

c932fb8

Format

941c4be

Lockfile

f997ca5

Fix type

92c3312

Merge branch 'main' into content-layer-loader

e0bd9cd

Handle loading for data store JSON

5cd5670

Merge branch 'main' into content-layer-loader

6ca356a

Use rollup util to import JSON

3fcdc59

Fix types

14322d4

Merge branch 'main' into content-layer-loader

3e21f03

Format

8d310af

ascorbic requested a review from ematipico June 27, 2024 06:29

github-actions bot added the pkg: astro Related to the core `astro` package (scope) label Jun 27, 2024

ascorbic commented Jun 27, 2024

View reviewed changes

packages/astro/tsconfig.json Outdated Show resolved Hide resolved

ascorbic force-pushed the loader-types branch from 9087c09 to 5b8fad4 Compare June 27, 2024 06:41

feat: add typegen for loaders

d8f2d6e

ascorbic force-pushed the loader-types branch from 5b8fad4 to d8f2d6e Compare June 27, 2024 07:31

Change back to direct zod import

527073d

Merge branch 'content-layer' into content-layer-loader

0a89f15

Merge branch 'content-layer' into content-layer-loader

51fcda3

florian-lefebvre reviewed Jun 27, 2024

View reviewed changes

ascorbic added 2 commits June 27, 2024 11:12

Add tests

9de133f

Merge branch 'content-layer-loader' into loader-types

21a7c37

matthewp approved these changes Jun 27, 2024

View reviewed changes

Changes from review

6bb89bc

ascorbic force-pushed the content-layer-loader branch from e93ad3e to 6bb89bc Compare June 27, 2024 14:20

Merge branch 'content-layer-loader' into loader-types

47649ca

Base automatically changed from content-layer-loader to content-layer June 28, 2024 08:00

An error occurred while trying to automatically change base from content-layer-loader to content-layer June 28, 2024 08:00

Merge branch 'content-layer' into loader-types

e98c6d5

ascorbic merged commit f9a3998 into content-layer Jun 28, 2024
14 checks passed

ascorbic deleted the loader-types branch June 28, 2024 08:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add typegen for loaders #11358

feat: add typegen for loaders #11358

ascorbic commented Jun 27, 2024

changeset-bot bot commented Jun 27, 2024 •

edited

Loading

ascorbic Jun 27, 2024

ematipico commented Jun 27, 2024

ascorbic commented Jun 27, 2024

ematipico commented Jun 27, 2024 •

edited

Loading

ascorbic commented Jun 27, 2024 •

edited

Loading

ematipico commented Jun 27, 2024

ascorbic commented Jun 27, 2024

florian-lefebvre Jun 27, 2024

feat: add typegen for loaders #11358

feat: add typegen for loaders #11358

Conversation

ascorbic commented Jun 27, 2024

Changes

Testing

Docs

changeset-bot bot commented Jun 27, 2024 • edited Loading

⚠️ No Changeset found

ascorbic Jun 27, 2024

Choose a reason for hiding this comment

ematipico commented Jun 27, 2024

ascorbic commented Jun 27, 2024

ematipico commented Jun 27, 2024 • edited Loading

ascorbic commented Jun 27, 2024 • edited Loading

ematipico commented Jun 27, 2024

ascorbic commented Jun 27, 2024

florian-lefebvre Jun 27, 2024

Choose a reason for hiding this comment

changeset-bot bot commented Jun 27, 2024 •

edited

Loading

ematipico commented Jun 27, 2024 •

edited

Loading

ascorbic commented Jun 27, 2024 •

edited

Loading