regexs general utilities

[poyoho]

Clear and concise description of the problem

In vite, in order to give users a better development experience, we didn't convert all JS into ast for lexical analysis. There are some specific syntaxes that are implemented through string matching.

We will encounter the following boundary problems.

For example:

1. regexp need to ignore comments/strings

• in workerImportMetaUrl plugin

const importMetaUrlRE = /\bnew\s+(Worker|SharedWorker)\s*\(\s*(new\s+URL\s*\(\s*('[^']+'|"[^"]+"|`[^`]+`)\s*,\s*import\.meta\.url\s*\))/g

• in assetImportMetaUrl plugin

const importMetaUrlRE = /\bnew\s+URL\s*\(\s*('[^']+'|"[^"]+"|`[^`]+`)\s*,\s*import\.meta\.url\s*,?\s*\)/g

• in optimizer/scan

const scriptModuleRE =
  /(<script\b[^>]*type\s*=\s*(?:"module"|'module')[^>]*>)(.*?)<\/script>/gims
const scriptRE = /(<script\b(?:\s[^>]*>|>))(.*?)<\/script>/gims
const importsRE = /(?<!\/\/.*)(?<=^|;|\*\/)\s*import(?!\s+type)(?:[\w*{}\n\r\t, ]+from\s*)?\s*("[^"]+"|'[^']+')\s*(?=$|;|\/\/|\/\*)/gm

but ignored data is also needed (/* @vite-ignore */ is needed.)

// new Worker(new Url('./worker.js', import.meta.url))

/* new Worker(new Url('./worker.js', import.meta.url)) */

new Worker(new Url('./worker.js', import.meta.url), /* @vite-ignore */, {type: variable })

---

2. in html plugin

• todo

const inlineImportRE = /\bimport\s*\(("[^"]*"|'[^']*')\)/g

it will got error in this case (#6525)

System.import("/some/other/module.js")

[patak-dev]

I think we may need to see it in a PR to check how good the abstraction is at cleaning up the code. One detail to take into account is that in some cases we want to first do a clean up and then re-use the cleaned code through several operations. Maybe bundling the two stages in a single operation isn't the best. But it may be good for some cases.

I was also thinking on some basic utils like withEmptyComments(code) and withEmptyStrings(code)

Maybe we could also start using https://github.com/unjs/mlly so we can share the maintenance of some of these regexes.