Refactor: Isomorphic

[antfu]

Following this RFC: #91

DEMO: https://codepen.io/antfu/pen/zYKNeWN

Try out now: npm i @antfu/shiki or https://unpkg.com/@antfu/shiki

Changes

• Packages shiki-languages and shiki-themes get deprecated, languages and themes is now shipped with the shiki package
• Use rollup to build esm,cjs,iife bundle
• getHighlighter() internal changed from class to functional
  • New options themes allowing to load multiple themes asynchronously on initialization and being able to use them synchronously later.
  • renderToHTML() and codeToThemedTokens() support a third argument for specify the theme
  • New async loadLanguage() and async loadTheme() to load the resources incrementally
  • New getBackgroundColor() and getForegroundColor() API.
• CDN Support (iife build)
  • Exported to global variable shiki
  • shiki-renderer-svg will extend the shiki variable and available at async shiki.getSVGRenderer()
  • Builds for UNPKG and jsDelivr work out-of-box, and can be used for any CDN with setCDN()
• Use lerna publish to do the version bumping and release (all packages will be the same version)
• Removed plist support as they are already normalized to json

BREAKING CHANGE

• codeToThemedTokens() options moved from the third arg to fourth, leaving the third for specifying theme
• when langs option in getHighlighter() is specified, it will no longer include all the bundled languages, use langs: [...BUNDLED_LANGUAGES, myCustomLanguage] to have the previous behavior
• loadTheme is now async

[orta]

I've given this a clone locally and run through a lot of the changes.

I think this is really well thought PR, and we should merge it. I don't know how free @octref is, so I will give this at least till the end of January for his feedback. If we don't get any, I'll merge this, move the TypeScript website to it and if that all goes well then I'll ship the npm module.

After this PR, shiki should ideally be at v0.9.x - with shiki in the browser as doable then I don't know what else should be blocking hitting 1.0 and getting it well documented for others to use.

[43081j]

Can you add exports to your package manifests? Things are moving away from module and what not. Webpack also supports this, and possibly rollup, so it could be good to have it specified.

Like here:
https://nodejs.org/api/packages.html#packages_conditional_exports
https://webpack.js.org/guides/package-exports/

Also, what if I don't want to use a CDN? do you have an example of the usage? is it expected we would load the file ourselves and pass the resulting object in for a theme for example?

Could you possibly allow us to pass a path in to a theme/lang which the loader can then deal with?

edit: looks like it will handle being passed a path but will warn about the CDN being missing if i read it right

edit2: you also rely on path, what happens with this in the browser? do you have the bundler strip it?

[43081j]

presumably this means if we want to load a theme by name, it must always exist in themes/ which often won't be true. can it be changed to allow us to specify the exact path ourselves?

[antfu]

You can call the loadTheme API first and then passed the resolved theme object to it. You can also use your own loader to load your grammar/theme json and pass them to shiki, this is how you use it without CDN.

[43081j]

do you not think it would be a quick win to allow that to be deal with by the loader too? you already have the logic for CDNs and it would be identical logic almost for your own paths. it'd save consumers having to write pretty much the same logic in their codebases. just a thought.

or even allow themes/ to be some configurable root path, and the same for the others

[antfu]

Not sure if I got your points, can you show me some examples? Thanks.

[43081j]

this is making a strong assumption that themes and languages always exist at themes/ and whatever directory you ship languages in.

but this isn't true for many consumers due to bundling. it is increasingly likely that projects will split bundles and use dynamic imports these days, which means its possible the themes may well exist in production as individual files but the bundle which loads them may no longer be relatively in the same place it was in this repo.

so my suggestion would be one of the following:

// ideal but difficult for you because you wouldn't know which string is a path, and which is a theme name except by pattern
loadTheme('./my-own-theme.json');

// alternatively, make the assumed path configurable
registry.themesRoot = '/foo/bar';
loadTheme('test'); // results in '/foo/bar/test.json'

of course where themesRoot exists in config, so wherever you initialise shiki.

otherwise, this means your current code is fine but anyone who doesn't ship this package exactly as-is will have to manage loading by themselves (of assets). which seems a shame given how small an improvement it may be

[43081j]

for example, one of our main repos uses webpack but does not use non-standard imports, i.e. it sticks to native supported imports. which means no magical webpack imports (like json imports, etc).

in our case, we would ship the languages we want in an assets directory somewhere and load them up from the bundle which will be elsewhere.

in that case we'd have to deal with the loading manually to avoid your hard coded themes/ path. so it would be nice to just tell shiki where to find the themes, or just give a path to the theme we want.

[antfu]

Yeah, that makes sense. We can have a logic to detect if it's a built-in theme or relative/absolute path. While I don't have much bandwidth on this atm, would you mind opening a PR against my fork so the changes can be reflected here? Thanks.

[43081j]

No worries, I'll have a look through it soon and see what I can come up with

[43081j]

out of interest, how does this work in browsers? is rollup or something stripping it?

[43081j]

One last note is that your ESM output will not be consumable by browsers, which is a shame seeing as that is the way forward, not IIFEs.

If you could output an actual ESM bundle, that would be so nice. It wouldn't be a big change to your rollup file, it just means bundling it with all externals. So we can do this:

<script type="module">
import {getHighlighter} from 'shiki'; // assuming we setup an import map already
// do stuff
</script>

this is also where exports would come in useful (in package.json) and means we can consume shiki in web components (which is why im here in the first place 😂 )

[antfu]

Agreed that we can have a browser ESM build

[orta]

@octref one last ping, I'll give this one last look over next week (deal with conflicts etc) and merge otherwise

[43081j]

FYI i did open a PR against this PR with the suggested path option at antfu#1

that and a browser ES module (it could be a bundle still, just in esm format) would be pretty useful to have, if possible

[antfu]

Oh sorry, @43081j I didn't see it. Somehow GitHub doesn't watch for forks automatically. I will have a look later and get back to you.

Will solve the conflicts as well. (the main branch has a GitHub action that fetches the file updates which cause the conflicts, shouldn't be a big problem tho)

[43081j]

no worries, if you have a better approach in mind just let me know and ill rework it if needed 👍

[orta]

Alright, tests pass and things seem to build fine - I've given the diff another read over and I think this is good to go.

[43081j]

@antfu did this end up containing a browser esm build? or will it have to be done in another PR?