Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve js engine support (and run tests) in pre-ES2024 envs #847

Merged
merged 1 commit into from
Nov 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/guide/regex-engines.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ The JavaScript engine is best when running in the browser and in cases when you

For the best result, [Oniguruma-To-ES](https://github.com/slevithan/oniguruma-to-es) uses the [RegExp `v` flag](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/unicodeSets), which is available in Node.js v20+ and ES2024 ([Browser compatibility](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/unicodeSets#browser_compatibility)).

For older environments, it can use the `u` flag but somewhat fewer grammars are supported.
For older environments, it can use the `u` flag but this results in a few less grammars being supported.

By default, the runtime target is automatically detected. You can override this behavior by setting the `target` option:

Expand Down
8 changes: 4 additions & 4 deletions docs/references/engine-js-compat.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Compatibility reference of all built-in grammars with the [JavaScript RegExp engine](/guide/regex-engines#javascript-regexp-engine-experimental).

> Generated on Friday, November 22, 2024
> Generated on Sunday, November 24, 2024
>
> Version `1.23.1`
>
Expand All @@ -20,7 +20,7 @@ Compatibility reference of all built-in grammars with the [JavaScript RegExp eng
## Supported Languages

Languages that work with the JavaScript RegExp engine, and will produce the same result as the WASM engine (with the [sample snippets in the registry](https://github.com/shikijs/textmate-grammars-themes/tree/main/samples)).
In some edge cases, it's not guaranteed that the the highlighting will be 100% the same. If that happens, please create an issue with the sample snippet.
In some edge cases, it's not guaranteed that the highlighting will be 100% the same. If that happens, please create an issue with the sample snippet.

| Language | Highlight Match | Patterns Parsable | Patterns Failed | Diff |
| ------------------ | :-------------- | ----------------: | --------------: | ---: |
Expand Down Expand Up @@ -210,7 +210,7 @@ In some edge cases, it's not guaranteed that the the highlighting will be 100% t
| wenyan | ✅ OK | 18 | - | |
| wgsl | ✅ OK | 44 | - | |
| wikitext | ✅ OK | 104 | - | |
| wolfram | ✅ OK | 501 | - | |
| wolfram | ✅ OK | 501 | - | 10 |
| xml | ✅ OK | 169 | - | |
| xsl | ✅ OK | 171 | - | |
| yaml | ✅ OK | 48 | - | |
Expand Down Expand Up @@ -254,7 +254,7 @@ Languages that throw with the JavaScript RegExp engine, either because they cont
| rst | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=rst) | 1855 | 2 | 62 |
| hack | ❌ Error | 947 | 1 | 114 |
| purescript | ❌ Error | 72 | 1 | 42 |
| cpp | ❌ Error | 510 | 2 | 27 |
| cpp | ❌ Error | 510 | 2 | 8 |
| csharp | ❌ Error | 306 | 3 | 204 |
| markdown | ❌ Error | 115 | 3 | 857 |
| swift | ❌ Error | 326 | 3 | 40 |
Expand Down
5 changes: 1 addition & 4 deletions packages/engine-javascript/test/compare.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -142,9 +142,7 @@ const cases: Cases[] = [
},
]

describe.skipIf(
+process.versions.node.split('.')[0] < 20,
)('cases', async () => {
describe('cases', async () => {
await loadWasm(import('@shikijs/core/wasm-inlined'))

const resolved = await Promise.all(cases.map(async (c) => {
Expand All @@ -163,7 +161,6 @@ describe.skipIf(
const engineWasm = createWasmOnigLibWrapper()
const engineJs = createJavaScriptRegexEngine({
forgiving: true,
target: 'ES2024',
})

const shiki1 = await createHighlighterCore({
Expand Down
4 changes: 1 addition & 3 deletions packages/engine-javascript/test/general.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@ import { describe, expect, it } from 'vitest'
import { createHighlighter } from '../../shiki/src/index'
import { createJavaScriptRegexEngine } from '../src'

describe.skipIf(
+process.versions.node.split('.')[0] < 20,
)('should', () => {
describe('should', () => {
it('works', async () => {
const shiki = await createHighlighter({
themes: ['vitesse-light'],
Expand Down
5 changes: 1 addition & 4 deletions packages/shiki/test/core-sync.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,7 @@ import { createHighlighterCoreSync } from '../src/core'
import js from '../src/langs/javascript.mjs'
import nord from '../src/themes/nord.mjs'

describe.skipIf(
// JavaScript engine requires Node v20+
+process.versions.node.split('.')[0] < 20,
)('should', () => {
describe('should', () => {
const engine = createJavaScriptRegexEngine()

it('works', () => {
Expand Down
12 changes: 6 additions & 6 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion pnpm-workspace.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ catalog:
minimist: ^1.2.8
monaco-editor-core: ^0.52.0
ofetch: ^1.4.1
oniguruma-to-es: 0.6.0
oniguruma-to-es: 0.6.1
picocolors: ^1.1.1
pinia: ^2.2.6
pnpm: ^9.13.2
Expand Down
2 changes: 1 addition & 1 deletion scripts/report-engine-js-compat.ts
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ async function run() {
'## Supported Languages',
'',
'Languages that work with the JavaScript RegExp engine, and will produce the same result as the WASM engine (with the [sample snippets in the registry](https://github.com/shikijs/textmate-grammars-themes/tree/main/samples)).',
'In some edge cases, it\'s not guaranteed that the the highlighting will be 100% the same. If that happens, please create an issue with the sample snippet.',
'In some edge cases, it\'s not guaranteed that the highlighting will be 100% the same. If that happens, please create an issue with the sample snippet.',
'',
createTable(reportOk),
'',
Expand Down
Loading