module transform api design (Analysis)

[hardfist]

Purpose

There are five mainstream kinds of module transform api frontend tooling is using

• loaders(webpack) https://webpack.js.org/contribute/writing-a-loader/
• resolve + load + transform(rollup|vite): https://rollupjs.org/guide/en/#transform
• resolver + transformer (parcel) : https://parceljs.org/plugin-system/resolver/ https://parceljs.org/plugin-system/transformer/
• middleware style loader api(nodejs) : https://github.com/nodejs/loaders
• resolve + load + namespace : https://esbuild.github.io/plugins/#on-resolve

all of these ways have some pros and cons and we need to discuss which api fit us well

Core Concept

core concept for further discussion( I just borrow it from nodejs loader

• resolve: convert an module id to an URL to be loaded
• load: take an resolved URL from resolve hook and returns runnable code(JavaScript,Wasm,etc) as well as bundler's internal translator
• translator: bundler's internal loader like
  • json
  • css
  • base64
  • ...etc
• transform: convert code to another code (like convert css to javascript)
• user_land plugin: plugin implemented by developer out of bundler
• internal plugin: plugin implemented inside bundler

Use Cases

We need to handle the following scenario very well by loader plugin

• plugin coordinate
• virtual module support
• default cache
• high performance sourcemap

Let's discuss these in separate sections

[hardfist]

Plugin Coordinate

Convert module type

transform some file type to internal translator, such as convert an svg to jsx

esbuild: return loader in transform hook

transform(code,id){
   if(!id.endsWith('.svg')){
      return;
   }
  const jsxCode = svg2jsx(code);
  return {
     code: jsxCode,
     loader: 'jsx'
  }
}

rollup | vite mock query to file extension

method 1

• do all things in one transform hook https://github.com/gregberge/svgr/blob/main/packages/rollup/src/index.ts
• problem: not composable, we should leave jsx transform to jsx loader other than deal it in svg loader it self

transform(code,id){
   const { filepath, query} = parseRequest(id);
   if(!id.endsWith('.svg')){
      return;
   }
  const  jsxCode = svg2jsx(code);
  const jsCode = babel.transform(jsxCode); // we may not know what other transform doing to jsxCode, such as babel polyfill
  return {
    code: jsCode,
  }
}

method 2

• use virtual module proxy to delegate to other loader
• very hack and error prone
• hard to deal with loader query and file extension together

  {
      name: 'svg',
      async resolveId(id, importer) {
        // convert .svg to .svg&lang.jsx so jsx plugin can handle this
        if (id.endsWith('.svg')) {
          const { id: resolvedId } = await this.resolve(id, importer, { skipSelf: true });
          return {
            id: appendQuery(resolvedId, { loader: 'jsx' }),
          };
        } else {
          return;
        }
      },
      async transform(code, id) {
        const { filePath, query } = parseRequest(id);
        if (filePath.endsWith('.svg')) {
          const jsxCode = await transform(code);
          return {
            code: jsxCode,
          };
        } else {
          return undefined;
        }
      },
      async load(id) {
        const { filePath, query } = parseRequest(id);

        return {
          code: fs.readFileSync(filePath, 'utf-8'),
        };
      },
    },
    {
      name: 'jsx',
      async transform(code, id) {
        const loader = defaultLoader(id);
        if (loader === 'jsx') {
          const jsCode = esbuild.transformSync(code, { loader: 'jsx' }).code;
          return {
            code: jsCode,
          };
        }
      },
    },

The fundamental problem with transform hook is that it lacks support change loader type, which makes transform composable very hard

webpack loader

{
   test: /\.svg/,
   use: ['svgr-loader', 'babel-loader']
}

• very elegant, we only to setup loader chain

nodejs loader

• use koa middleware mode to handle loader chain

/**
 *
 * @param {string} url
 * @param {any} context
 * @param {Function} next
 * @returns
 */
async function load(url, context, next) {
  if (url.endsWith('.svg')) {
    let result = await next(
      url,
      {
        ...context,
        format: 'module',
      },
      next
    );
    const jsxCode = await transform(result.source.toString(), {}, {});
    const code = (
      await esbuild.transformSync(jsxCode, {
        loader: 'jsx',
      })
    ).code;
    return {
      format: 'module'
      source: code,
      shortCircuit: true,
    };
  } else {
    return next(url, context, next);
  }
}
/**
 *
 * @param {string} id
 * @param {*} context
 * @param {*} next
 */
async function resolve(id, context, next) {
  // handle unknown file extension problem
  if (id.endsWith('.svg')) {
    let url = new URL(id, context.parentURL).href;
    return {
      url,
      format: 'module',
      shortCircuit: true,
    };
  }
  return next(id, context, next);
}

async function load(url, context, next) {
  if (context.format === 'jsx') {
    const code = await next(url, { ...context, format: 'module' }, next);
    const jsCode = esbuild.transformSync(code, { format: 'esm' });
    return {
      source: jsCode.code,
      format: 'module',
      short,
    };
  } else {
    return {
      source: code,
      format: context.format,
    };
  }
}

[hardfist]

Vue Loader Implementation

Vue loader Implementation is a much more complex use case than svgr case, we need to deal with

• split vue file to five sections( script,template,style, custom block)
• each section may have its own preprocessor | postprocessor( jade、less、sass、ts、jsx and so on)
• every section in vue file need to go through multi type change, such as sass sction in .vue -> .sass -> .css

Parcel Implementation

• transform register

{
     "*.vue": ["@parcel/transformer-vue"],
    "template:*.vue": ["@parcel/transformer-vue"],
    "script:*.vue": ["@parcel/transformer-vue"],
    "style:*.vue": ["@parcel/transformer-vue"],
    "custom:*.vue": ["@parcel/transformer-vue"],
}

• transformer

const { Transformer } = require('@parcel/plugin');
const compiler = require('@vue/compiler-sfc');
const semver = require('semver');
const { hashObject } = require('@parcel/utils');
const path = require('path');
module.exports = new Transformer({
  canReuseAST({ ast }) {
    return ast.type === 'vue' && semver.satisfies(ast.version, '^3.0.0');
  },
  async parse({ asset }) {
    let code = await asset.getCode();
    let parsed = compiler.parse(code, {
      sourceMap: true,
      filename: asset.filePath,
    });
    let descriptor = parsed.descriptor;
    let id = hashObject({
      filePath: asset.filePath,
      source: code,
    }).slice(-6);
    return {
      type: 'vue',
      version: '3.0.0',
      program: {
        ...descriptor,
        script: compiler.compileScript(descriptor, {
          id,
          isProd: false,
        }),
      },
    };
  },
  async transform({ asset, options, resolve, config }) {
    const ast = (await asset.getAST()).program;
    console.log('ast:', ast);
    const { template, script, style, customBlocks } = ast;
    const code = await asset.getCode();
    let basePath = path.basename(asset.filePath);
    console.log('pipeline:', asset.pipeline);

    if (asset.pipeline) {
      // script:xxx.vue, template:xxx.vue, style:xxx.vue, custom:xxx.vue goes here
      switch (asset.pipeline) {
        case 'script':
          console.log('xxx:', script);
          const lang = script.lang;
          // pass script to 'ts'|'js'|'jsx'|'tsx' handler
          return [
            {
              type: lang || 'js',
              content: script.content,
            },
          ];
        case 'style':
          return [
            {
              type: style.lang || 'css',
              content: style.content,
            },
          ];
        case 'template':
          const template_code = compiler.compileTemplate({
            filename: asset.filePath,
            source: template.content,
            isProd: false,
            isFunctional: false,
          });
          return [
            {
              type: 'js',
              content: template_code,
            },
          ];
      }
    } else {
      // xxx.vue goes here
      return [
        {
          type: 'js',
          content: `
        require('script:./${basePath}');
        require('template:./${basePath}');
        require('style:./${basePath}');
        require('custom:./${basePath}');
        `,
        },
      ];
    }
  },
});

vite implementation

https://github.com/vitejs/vite/blob/main/packages/plugin-vue/src/index.ts#L171

Webpack implementation

using matchResource to deal with change module type in loader suggested by @sodatea
https://webpack.js.org/api/loaders/#inline-matchresource
webpack will support virtualResource to further simplify change module type webpack/webpack#15459
https://github.com/hardfist/loader-design/blob/main/webpack_loader/vue.js

const compiler = require('@vue/compiler-sfc');
const loader_utils = require('loader-utils');
const qs = require('qs');
const descCache = new Map();
const util = require('util');

/**
 * @this {import('webpack').LoaderContext<any>}
 */
module.exports = function vueLoader(content) {
  const callback = this.async();
  const inner = async () => {
    const parsed = compiler.parse(content, {
      sourceMap: false,
      filename: this.resourcePath,
    });
    const descriptor = parsed.descriptor;
    const { script, scriptSetup, styles, template, customBlocks } = descriptor;
    descCache.set(this.resource, {
      ...parsed.descriptor,
      script: compiler.compileScript(descriptor, {
        id: Math.random().toString(),
        isProd: false,
      }),
    });
    const query = qs.parse(this.resourceQuery, { ignoreQueryPrefix: true });
    if (query.vue === 'true') {
      let cache = descCache.get(this.resource);
      const { script, style } = cache;
      if (query.type === 'script') {
        return script.content;
      } else if (query.type === 'style') {
        const lang = styles[0].lang;
        return styles[0].content;
      }
    } else {
      const jsPath = this.resourcePath + '.js' + '!=!' + this.resourcePath + '?vue=true&type=script';
      // const jscode = await util.promisify(this.loadModule)(jsPath);
      const cssPath = this.resourcePath + '.less' + '!=!' + this.resourcePath + '?vue=true&type=style';
      // const csscode = await util.promisify(this.loadModule)(cssPath);
      // console.log('csscode:', csscode);
      return `require(${JSON.stringify(jsPath)});require(${JSON.stringify(cssPath)})`;
    }
  };
  return util.callbackify(inner)((err, data) => {
    callback(err, data);
  });
};

[hardfist]

virtual module

sometime we need to use virtual module to generate module on the fly, but it often cause some trouble for plugin.
there are different ways to do that.

special id

see https://github.com/vitejs/vite/blob/main/packages/plugin-vue/src/index.ts#L162

• use resolve hook to intercept virtual module id to avoid default resolver exception
• use load hook to return inline code
• problem: if we insert an plugin before vue plugin, we need to deal with virtual module id carefully, otherwise resolve hook and load hook may fail

const EXPORT_HELPER_ID = 'plugin-vue:export-helper'

 const helperCode = `
export default (sfc, props) => {
  const target = sfc.__vccOpts || sfc;
  for (const [key, val] of props) {
    target[key] = val;
  }
  return target;
}`;

export default function vuePlugin(){
  return {
    async resolveId(id){
        if(id === EXPORT_HELPER_ID) {
             return id
        }
    },
    load(id){
        if( id === EXPORT_HELPER_ID){
            return helperCode;
        }
    }
  }
}
`

id with query

we may add special query to annotate an module as virtual module，

• more compatible with default resolver
• easy to compat then special id(use parseRequest)
• pollute url and other plugin still need to handle polluted url

export default function vuePlugin(){
  return {
    async resolveId(id){
        if(parseRequest(id).query.vue){
          return id;
       }
    },
    load(id){
        if(parseRequest(id).query.vue){
            // do something
      }
    }
  }
}

rollup's id is easy to be broken when deal with polluted id, every plugin should do the parseRequest trick to avoid being broken,
webpack's loader do it better, it provides resource | resourcePath | resourceQuery parameters to loader, and automatic split query from url, so it's less broken then rollup

use data: to handle virtual module

• compatible with bundler
• It's ecmascript spec

import 'data:text/javascript;charset=utf-8;base64,Y29uc29sZS5sb2coJ2lubGluZSAxJyk7';
import {
  number,
  fn,
} from 'data:text/javascript;charset=utf-8;base64,ZXhwb3J0IGNvbnN0IG51bWJlciA9IDQyOwpleHBvcnQgY29uc3QgZm4gPSAoKSA9PiAiSGVsbG8gd29ybGQiOw==';

use virutalFS

webpack virtual module supports virtual module by customize its underly filesystem https://github.com/sysgears/webpack-virtual-modules/blob/master/src/index.ts#L193

use special namespace

• compatible with default resolver
• still need to deal with namespace conflict

export default function vuePlugin(){
  return {
    async resolveId(id){
        if(id === EXPORT_HELPER_ID) {
             return { path: id, namespace: 'virtual'}
        }
    },
    load({id,namespace}){
        if(namespace === 'virtual'){
            return helperCode;
        }
    }
  }
}

[kdy1]

You can restrict scope like https://github.com/kdy1/jab/blob/3ffc6f694e7a485296dfebb0cc4be78eee424a29/crates/jab_core/src/resource.rs

[hardfist]

so you are suggesting generate id use some id generator exposed by bundler.

const idGen = require('bundler/utils');
const EXPORT_HELPER_ID = idGen();

 const helperCode = `
export default (sfc, props) => {
  const target = sfc.__vccOpts || sfc;
  for (const [key, val] of props) {
    target[key] = val;
  }
  return target;
}`;

export default function vuePlugin(){
  return {
    async resolveId(id){
        if(id === EXPORT_HELPER_ID) {
             return id
        }
    },
    load(id){
        if( id === EXPORT_HELPER_ID){
            return helperCode;
        }
    }
  }
}
`

[kdy1]

Yeah, it's similar but actually I was thinking rust code

[hardfist]

but it doesn't solves the problem about plugin conflict. if we have another plugin before vuePlugin, how the plugin deal with the virtual module id.

export default function otherPlugin(){
  return {
    async resolveId(id){
       
    },
    load(id){
       const content = fs.readFileSync(id); // it will corrupt, if it didn't take care of the EXPORT_HELPER_ID, but how to deal with it
    }
  }
}
`

[kdy1]

I was not talking about resolveId + load case

[kdy1]

I found a problem of resolve + load + transform model.

In this model, transform pass should know the correct way to extract asset (like css) from the input file.
Otherwise, the loader should apply the transform.

[hardfist]

yes, we met lots of problems when using resolve + load + transform model, even it's hackable but it's not ideal
vite actually did some tricks to implment css extract by collect css module info in transform hook and manually concat css module file into extract file see https://github.com/vitejs/vite/blob/main/packages/vite/src/node/plugins/css.ts#L442