Core: Support new indexer API

[JReinhold]

Works on #23457

What I did

This PR adds support for the new story indexer API in StoryIndexGenerator. It doesn't actually convert all the existing indexers yet to the new API (except for in tests), that is a follow up PR.

How to test

Create a sandbox (or use the canary release listed below) and replace the current deprecated indexers with new ones, by adding this to main.js:

import fs from 'fs-extra';
import { loadCsf } from '@storybook/csf-tools';
...

  ...
  storyIndexers: () => [],
  indexers: async (existingIndexers) => {
    return [
      {
        test: /\.stories\.(m?js|ts)x?$/,
        index: async (fileName, options) => {
          const code = (await fs.readFile(fileName, 'utf-8')).toString();
          const csf = loadCsf(code, { ...options, fileName }).parse();
      
          // eslint-disable-next-line no-underscore-dangle
          return Object.entries(csf._stories).map(([key, story]) => ({
            key,
            id: story.id,
            name: story.name,
            title: csf.meta.title,
            importPath: fileName,
            type: 'story',
            tags: story.tags ?? csf.meta.tags,
          }));
        },
      },
      {
        test: /\.stories\.mdx$/,
        index: async (fileName, opts) => {
          let code = (await fs.readFile(fileName, 'utf-8')).toString();
          const { compile } = await import('@storybook/mdx2-csf');
          code = await compile(code, {});
          const csf = loadCsf(code, { ...opts, fileName }).parse();
      
          // eslint-disable-next-line no-underscore-dangle
          return Object.entries(csf._stories).map(([key, story]) => ({
            key,
            id: story.id,
            name: story.name,
            title: csf.meta.title,
            importPath: fileName,
            type: 'story',
            tags: story.tags ?? csf.meta.tags,
          }));
        },
      },
      ...(existingIndexers || []),
    ];
  },

Checklist

• Make sure your changes are tested (stories and/or unit, integration, or end-to-end tests)
• Make sure to add/update documentation regarding your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts
• Make sure this PR contains one of the labels below.

["cleanup", "BREAKING CHANGE", "feature request", "bug", "build", "documentation", "maintenance", "dependencies", "other"]

🦋 Canary release

This pull request has been released as version 0.0.0-pr-23626-sha-0d5a537f. Install it by pinning all your Storybook dependencies to that version.

More information

Published version	0.0.0-pr-23626-sha-0d5a537f
Triggered by	@JReinhold
Repository	storybookjs/storybook
Branch	support-new-indexer
Commit	0d5a537f
Datetime	Thu Jul 27 10:47:15 UTC 2023 (1690454835)
Workflow run	5679592639

To request a new release of this pull request, mention the @storybookjs/core team.

core team members can create a new canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=23626

[JReinhold]

You could have both the deprecated storyIndexers and the new indexers trying to index the same file, albeit it's a bit unlikely. In this scenario I've currently chosen that the new indexers takes precedence and the deprecated storyIndexers would be ignored for such a file (thus only a single index entry is generated), but I'm open to other suggestions, such as throwing an error instead.

[JReinhold]

these two indexers below are very likely how they will be implemented for real in a follow up PR

[JReinhold]

it's unfortunate that we have to rely in the "internal" _stories field here, but that's the only way to get stories' keys.
I could also change CsfFile.stories to return the object instead of only the values, but I don't know if that would be considered a breaking change? Depends on if we consider CsfFile a public API or not.

[tmeasday]

Why do we need the key here in this case if we already have a story.id? Or is that not guaranteed to exist?

[JReinhold]

key is required in the new IndexInput. And key and id is not the same, here's an example:

{
	key: 'LoggedIn',
	id: 'example-header--logged-in',
	name: 'Logged In',
  title: 'Example/Header',
}

[tmeasday]

@JReinhold but in the example above, does key do anything? Would it not just get thrown away? IIUC key is just used in preference to name when calculating id when id itself isn't passed.

[JReinhold]

You are right, key is just used to calculate name, name and title are used to calculate id.
So in this case the key is just to satisfy the typings that require a key.

We could make the types more clever by requiring key or name, but I think we need a longer discussion about key/name/id.

[JReinhold]

as we don't have a concept of component but only story entries here now, I don't see a way to get tags and title from the meta. I just opted to get it from the first entry.
If anyone can come up with a smarter way or a reason why this is bad, let me know.

[tmeasday]

Why do we need the key here in this case if we already have a story.id? Or is that not guaranteed to exist?

[tmeasday]

Does this handle "docs only" .stories.mdx files? I don't see tests for those so I am guessing maybe not?

[JReinhold]

It does not. I plan to add that in the follow up PR for implementing the indexers, since that logic now lives there instead.

[JReinhold]

none of the tests here are new, they've just been copied back from the deprecated test suite. The only thing new here is the indexers above

[kasperpeulen]

Minor nitpick, rest LGTM