How to integrate react-refresh in esbuild to implement hmr ?

[sessionboy]

I use esbuild to build my react application, no webpack, no rollup, only esbuild.It works well, everything looks great.
But I don't know how to implement hmr in esbuild.

   service.build({
      platform: 'browser',
      format: 'esm',
      splitting: true,
      bundle: true,
      loader: {
        '.js': 'jsx',
        '.png': 'file',
        '.svg': 'file',
        '.jpg': 'file'
      },
      minify: false,
      entryPoints: [
        path.join(paths.appSrc, '/layout/entry-client.tsx')
      ],
      outdir: path.join(paths.appBuild, 'client'),
      sourcemap: true,
      metafile: path.join(paths.appBuild, 'client-metafile.json'),
      define: {
        'process.env.NODE_ENV': JSON.stringify('development'),
        'process.browser': "true",
      }
    })   

I checked the hmr implementation of vite. But the difference is that vite only injects the HMR code when the module is requested, not during the build.So this is easy to implement.

I used code splitting.

I need to inject hmr code during the first build process. Re-execute the build when the code is updated, and update the code dynamically.

This is not a simple matter.

I guess esbuild plugin should be able to implement it, but I need some guidance. How can i implement it ?

[evanw]

Sorry, esbuild isn't designed for HMR. It's possible to use esbuild's transform API as part of another HMR tool (e.g. what Vite does) but this isn't something that esbuild's build API supports. I'm not sure if it can be done with an esbuild plugin because esbuild's plugin API deliberately doesn't let you substitute your own linker. I haven't ever implemented HMR before but I believe HMR requires a different type of linker. For example the react-refresh documentation says that integrating it requires your bundler to support HMR.

[sessionboy]

esbuild isn't designed for HMR.

I know this, I think hmr should be an additional feature of esbuild.
I don't want to use webpack or other bundler, I think the future will be no-bundler.
Many people expect esbuild to implement hmr. If esbuild implements hmr, then the no-bundler goal will be achieved immediately, this will make web go into the future.
Snowpack proposed the esm-hmr spec, maybe this can help you implement hmr.

What's bad is that react-refresh currently only supports babel's transform. I haven't found any documentation or case of using esbuild's transform to implement react-refresh. https://github.com/facebook/react/blob/master/packages/react-refresh/src/ReactFreshBabelPlugin.js

Look forward to esbuild's realization of HMR and lead us into the future.

Cheers 🍻.

[nojvek]

+1 on this. HMR support or webpack-dev-server like support for esbuild would be a god send. I’m willing to dig into the source code and work on some PRs if someone’s willing to guide and review PRs.

[zaydek]

So I took a close look at how Dan outlined integrating HMR: facebook/react#16604 (comment). At first it didn’t make any sense but then I started to get it.

Edit: Nevermind, I think you need to use Babel to pull this off. Based on what I do understand, this might be a significant undertaking for esbuild because it requires a lot of AST / source transformations which do seem beyond the scope of what esbuild is trying to do. esbuild is a bundler after all with loading and plugin capabilities. It’s not a transformer like Babel (which is what React Refresh relies on).

What I understood is this:

There’s a module you can require: require('react-refresh/runtime') that is responsible for most of the implementation. Still, your code needs to be React Refresh-aware. In order for that to happen, there’s basically a lot of boilerplate that you need to wrap every module with and your top-level ReactDOM.render code.

If you look closely here: https://github.com/vitejs/vite/blob/main/packages/plugin-react-refresh/index.js, the way Vite implements HMR (which uses esbuild for part of its toolchain) is that it uses Babel to transform the JS. This is actually pretty easy to understand conceptually once you understand how React Refresh works. It basically just wraps your app / modules with some glue-code.

You can see Evan You is literally wrapping code here: https://github.com/vitejs/vite/blob/main/packages/plugin-react-refresh/index.js#L123.

      return {
       code: `${header}${result.code}${footer}`,
       map: result.map
     }

In theory, someone should be able to make a proof-of-concept repo that demonstrates React Refresh without Babel -- just hardcode all of the boilerplate. See if that works. Then you have a pure implementation. From that, it’ll be easier to extrapolate how to apply these techniques to esbuild.

From what I grok, esbuild may be able to handle all of this today. From what I can tell, webpack is used in order to talk to Babel in order to a) get module.ids automatically and b) transform code. I’m not sure webpack is doing more than that (see https://github.com/pekala/react-refresh-test/blob/master/webpack.config.js).

[nojvek]

I have written some custom hot module reloaders with webpack, and there interface is fairly clean
https://webpack.js.org/concepts/hot-module-replacement/

The general idea is that you accept via module.hot.accept(moduleId, handlerFn) and inside handlerFn, do a require call to get latest instance of module and do some sort of updating the app and change the module.exports to the new module. If it is not accepted, it bubbles up the require tree until something accepts it.

e.g https://github.com/mixpanel/panel/blob/5014a48bc72d6ab710f35c81c9d7d841f32d9c85/hot/template-loader.js#L25-L29

[evanw]

The general idea is that you accept via module.hot.accept(moduleId, handlerFn) and inside handlerFn, do a require call to get latest instance of module and do some sort of updating the app and change the module.exports to the new module. If it is not accepted, it bubbles up the require tree until something accepts it.

Note that module.hot is undefined in esbuild, since esbuild doesn't implement HMR, so module.hot.accept() will crash at run-time.

I don't want to use webpack or other bundler, I think the future will be no-bundler.

You may not want to use esbuild then since esbuild is a bundler. You may want to consider using Snowpack or Vite instead. They are both no-bundler options that implement HMR and use esbuild internally for speed.

[nojvek]

I’ll probably have to investigate esbuild’s incremental build abilities. The whole idea of HMR and dev-server is that once you make a change in the editor, you immediately see it’s effect in the browser or similar runtime without having to rebuild the entire codebase or refresh the browser.

Incremental devloop being in milliseconds is a big deal. I wonder how snowpack gets around this. May be esbuild exposes some version of an incremental module update api?

[zaydek]

So I figured out how to implement esbuild incremental recompilation + server-sent events to trigger reloads which feels like HMR but works today. I used this playground to validate my ideas: https://github.com/zaydek/esbuild-dev-server.

I use a poll-based file watcher to look for changes to source files. The public dir is served and uses the /sse route to send sever-sent events. Those events allow me to trigger refreshes on the client browser tab. Build errors are passed along to the client browser tab and also logs them on the server.

It’s not HMR but I basically can’t tell the difference. I also prefer this implementation since it doesn’t care about React anymore (React Refresh is tightly coupled to React) and there’s no quasi states to worry about anymore because state is never persisted between reloads (unless you write to localStorage).

[osdevisnot]

@zaydek i did not dig into this yet, but how is esbuild-dev-server this different than https://github.com/osdevisnot/sorvor?

[zaydek]

@osdevisnot It’s functionally similar. I actually reviewed your code which helped me a lot.

I’ll put this in a <details> so I don’t distract from the original intent of this thread too much.

Main differences:

1. Your code uses a watcher package which implements poll-based watching more precisely whereas I just did it from scratch.
   Implementation here: https://github.com/zaydek/esbuild-dev-server/blob/master/watch.go#L141.
2. Besides that, I noticed your SSE implementation is more complicated than it needs to be, unless I’m missing something (which I probably am).
   SSE can be simplified to be as simple as this: https://github.com/zaydek/esbuild-dev-server/blob/master/server_sent_event.go. This should work for 5-6 browser tabs (the limitation is because of HTTP 1.1). I noticed you are using a lot of channels to orchestrate events but I don’t think it actually helps you, since SSE is an HTML specification and the burden of implementation lies with the browser. A SSE backend simply needs to set headers, write messages, and flush, that’s pretty much it.
3. esbuild-dev-server is just a demonstration for my own education. It’s not an end product whereas sørvør is actually designed to solve a problem.

Anyway, I needed to understand all of the moving parts from first principles in order to keep working on my page-based router: https://github.com/zaydek/retro/tree/a54bf320c7787774c61412bec4237c9c9b7f2d22. I’m trying to solve for a rapid-development environment with page-based routes vs. a single-page application architecture.

I went with poll-based watching (like you) because it’s the simplest and platform-independent by default. Rationale here: #21 (comment).

[jeremy-coleman]

There is https://github.com/progrium/hotweb , which is using esbuild and gorilla ws.

You dont actually need to implement module.hot api to get stateful reloading. Most hmr servers send a “reload” event via ws.

My experience with go is borderline none, but was looking through the esbuild docs/code and i think there is the foundation to do everything needed using the esbuild prelude and output meta.json . The wrappers esbuilds appends at the top of bundles, like __toModule(_import(“react”)) or whatever it is could be augmented a little bit to listen for reload events. The reload event could have a data object that has the changed module identifier(s) and the new code. The logic for re-executing the bundle could be wrapped around the original bundle using the prelude api. Is there a hook to intercept the final write ? Changing the bundles identifiers to use src paths would be super helpful

there is a very simple implementation here using browserify (esbuild is setup also, not originally for esbuild hmr, but its a decent starting point to do some hacking )
https://github.com/jeremy-coleman/esbuild-vs-omnify-r3f the hmr logic is in tools/omnify/plugins . There is also a simplified version here https://github.com/jeremy-coleman/prefresh-browserify-help-wanted/blob/main/tools/livereload/plugin-no-sourcemap.js . Its kind of wtf hard to follow bc its mapping module deps jsonish output to something different, but all that hackery could be probably done much easier using the esbuild output format. It changes the output to strings , which are instantiated in the browser using new function and eval’d to update the runtime. Definitely hacky but its very fast and will maintain state on reloads in everything but internal component state without needing react hot loader or anything(although adding it will give u hot internal component state updates too, which is cool). However, implementing a jsx transform for hot updates is probably out of the question. Sucrase is fast as f though, so maybe just use that if you need the hottest of hot. Anyway, to give an idea of capabilities.. given this react/preact component:

const Test = () =>
  useObserver(() => {
    return (
      <div>
        <header>App Headers</header>
        <h1 style={{ textAlign: "center" }}>Count: {model.counter}</h1>
        <input
          type="range"
          min={0}
          max={1000}
          value={model.counter}
          onInput={(evt) => (model.counter = Number((evt as any).target.value))}
          style={{ width: "100%" }}
        />
      </div>
    );
  });

Just as an example, you can update the text of App Headers and the range input slider will maintain state, because the value is held in a different module. This is without any hmr jsx transforms or anything, just the hot reload socket api. I think this is good enough for most apps. I think this level of reloading could be done via esbuild with very little code, i just have no idea how to do it.

[geordie2020]

@zaydek
Hi. I'm in a similar situation as you were, when your were writing this.

It’s not a transformer like Babel (which is what React Refresh relies on).

From what i understand, esbuild can indeed be used as a transformer like babel. Could you elaborate, on why it is not suitable for this situation?

[zaydek]

@zaydek
Hi. I'm in a similar situation as you were, when your were writing this.

It’s not a transformer like Babel (which is what React Refresh relies on).

From what i understand, esbuild can indeed be used as a transformer like babel. Could you elaborate, on why it is not suitable for this situation?

I can try to help based on my understanding of esbuild.

It's important to think of esbuild as a build-time tool and not a runtime tool. HMR is basically a clever way to layer a system on top of the runtime. So that means in most cases, HMR needs to know something about your runtime, say for example React. React Refresh or whatever Facebook uses is a React tool. esbuild is a JavaScript tool. In order for esbuild to support HMR, esbuild would need to interact with React in a way that it's not designed to. esbuild does understand JSX but that's mostly a build-time transformation.

So the point is, I think, more philosophical. esbuild is trying to be a compile-time tool for JavaScript, but it's not runtime or framework-aware in the way it would need to be to truly support HMR. HMR if I'm not mistaken is a deeply runtime / framework-specific tool, and esbuild is anything but that.

I hope that clears up any of the misconceptions. I'm not saying HMR isn't possible with esbuild via plugins or something like that, I'm just trying to help explain why esbuild doesn't particularly have an opinion on HMR to begin with.

[evanw]

Closing this issue because this is out of scope. I recommend using another tool instead.