Uncaught ReferenceError: Buffer is not defined

[thiagomajesk]

Hi! I'm having this weird error message using Vite as a bundler which seems to happen in a very specific scenario.

I couldn't reproduce it using the default vite dev server (ie.: npx vite), but I managed to create a small repro of the scenario I have in another project that uses Phoenix and does not use vite's dev server.

I can confirm though that the error only happens when the last import is present (@uppy/transloadit):

import Uppy from '@uppy/core'
import Dashboard from '@uppy/dashboard'
import Transloadit from '@uppy/transloadit'

---

Creating an simple HTML file and adding a script tag to the head works fine using vite's dev server (npx vite):

index.html

<script type="module">
  import Uppy from '@uppy/core'
  import Dashboard from '@uppy/dashboard'
  import Transloadit from '@uppy/transloadit'
<script>

But it doesn't work if I use vite build with my current config for some reason.
Here's a small repro that illustrates the problem: vite-repro.zip

Instructions:

• npm i --prefix assets
• npm run build --prefix assets
• npm run start --prefix assets (for the HTTP server).
• See error in Chrome dev tools (I'm using version 94.0.4606.81)

[Murderlon]

Hi, @thiagomajesk. As I initially expected, this is not an issue with Uppy but a bundler issue. I'm not familiar with bundlers so it left me confused for a bit so pardon me for the late reply. We don't use Buffer, but our dependency socket.io does. In Vite, Node build-ins are not shimmed. I would as a starting point suggest taking a look at this discussion on how to shim Buffer: vitejs/vite#3126

[thiagomajesk]

Hi @Murderlon, thanks a lot for looking into it.

I was in fact expecting that this could be a Vite issue because of how it serves modules as-is but wasn't sure... My tests were inconclusive because, here's the weird part: I've managed to make it work purely with vite serve and not with vite build (btw, have you checked that this in fact is working as expected?)

I also stumbled upon many discussions like vitejs/vite#3126 (comment) and vitejs/vite#728 (comment) and I don't know what I should make of that information when I read the following statements from the project maintainers:

I'll try to follow your advice on the shimming process, but in the meantime, If we can confirm that this is in fact related to Vite alone I'll proceed to open an issue on their repo so we can get investigate it further.

[Murderlon]

Hi, I don't think another issue in Vite will help here as it is the same problem as has been surfaced in multiple discussions, unless it's specifically around trying to work around this instance. We are considering patching socket.io with yarn patch in our repository to remove the code we don't use, which includes the Buffer usage, so it should also fix the build in Vite. But this is not sure yet so I wouldn't make it your first bet on resolving this.

Regarding the response from the maintainers from Vite, I agree with the sentiment, but it's a very unnuanced take as it's not always feasible to "swap it out with something more modern". We rely heavily on socket.io, and swapping it would be a risky procedure.

Lastly, socket.io uses Buffer to support binary messages, which is always included even if it's unused like in our project. So we could open an issue there if that could be resolved somehow.

[RobWalker]

I ran into this issue in a Vue 3 project, and the best workaround I found was to defer the loading of the components that relied on Uppy.

In my case all the Uppy code was within an ImageContentPanel component, so changing the usage of this component from

  import ImageContentPanel from '@editor/components/settings/ImageContentPanel.vue';

to

const ImageContentPanel = defineAsyncComponent(() =>
  import('@editor/components/settings/ImageContentPanel.vue')
);

Caused the production vite build to bundle those libraries separately ... and everything just worked.

I don't quite understand /why/ it works at runtime with this approach vs a single bundle since Buffer is still undefined.

As far as this Uppy is concerned to @Murderlon point ... it would be ideal if socket.io could be more selective even if only because it sounds like a bunch of unused code is getting loaded/executed which is never ideal in the browser.

[thiagomajesk]

@Murderlon Sorry for the delay...

I tried going the shimming route using rollup-plugin-polyfill-node but I kept having problems with it. Following the instructions on this thread does not seem to be enough to solve it.

import polyfillNode from 'rollup-plugin-polyfill-node'

export default {
  plugins: [polyfillNode()],
  optimizeDeps: { exclude: ["@uppy/*"]}
  // [...]
}

While building I get this error:

12: 
13: /* global window */
14: import { Base64 } from 'js-base64';
             ^
15: import URL from 'url-parse';
16: import DetailedError from './error';
error during build:
Error: 'Base64' is not exported by node_modules/js-base64/base64.js, imported by node_modules/tus-js-client/lib.esm/upload.js

---

Alternatively, I also tried making buffer available on window using these suggestions without success:

• How to shim Buffer and crypto? vitejs/vite#3126 (comment)
• Using (Node’s) Buffer in your polymorphic Vite apps vitejs/vite#2785
• Using (Node’s) Buffer in your polymorphic Vite apps vitejs/vite#2785 (comment)

One of the errors I get when I try to manually polyfill buffer:

Uncaught TypeError: Class extends value undefined is not a constructor or null

---

My impression so far is that It seems there isn't a straight-up solution for this or I'm solving the wrong problem. I'm also not a bundler expert, so I'm missing whether this could be considered just a bundling issue or a package delivery problem.

For what is worth, I've also searched a little more and it seems people using Webpack had similar problems because Webpack 5 does shim buffer and friends by default:

• https://webpack.js.org/migrate/5/#test-webpack-5-compatibility
• Buffer removed, it cause some trouble webpack/changelog-v5#10.

[lxsmnsyc]

@thiagomajesk I can confirm that this happened to us as well. Started from the Buffer error then when we tried polyfilling Buffer, it then raised the error regarding the missing "Writeable", which turned out, on further inspection, to be an empty module. I also tried using terser instead of esbuild for minification but to no avail. It's weird that the error only happens in build and not on dev.

At this point we'll probably ditch the Transloadit Plugin and look for alternatives as this broke the entire app we're working on.

[lxsmnsyc]

After some research, Writable seems to come from Node's stream module: https://github.com/websockets/ws/blob/26a46f33ffc74ad91d4adc7662489c79e64032c0/lib/receiver.js

edit:
Tried with rollup-plugin-polyfill-node and it still just made things worse. Looks like we're going to look for alternatives.

[Murderlon]

Thanks for the detailed updates. We are discussing this internally at the moment how to best solve this. I will post an update once we have something.

[RobWalker]

We're now running into this again at runtime: hitting the same issue as @thiagomajesk with Writable. I have no idea why it worked for a while in production builds and it now broken. Presumably something somewhere got updated.

Attempts to polyfill it have all fallen flat or lead down a rabbit hole of other errors.

It looks like it comes down to using vite or the transloadit plugin and we can't ditch vite easily, so we will also need to look for another option 😞

Definitely frustrating since the code itself works fine in a dev build, so the problem seems to be that too much code is getting run in production which is not what we want!

[lxsmnsyc]

It looks like it comes down to using vite or the transloadit plugin

This was one of the confusing parts. We have two apps, one that experiences this problem, the other doesn't and works seamlessly. Both uses the same code for uppy and its components. It's weird as to which of these should be approached to find the solution.

[thiagomajesk]

It looks like it comes down to using vite or the transloadit plugin and we can't ditch vite easily, so we will also need to look for another option 😞

@RobWalker I don't think Vite is the problem anymore. Since Webpack 5 doesn't do shims out of the box as well, I figure folks over there would have the same problems we are having. Even shimming with the same lib is not working for some reason.

We're now running into this again at runtime: hitting the same issue as @thiagomajesk with Writable. I have no idea why it worked for a while in production builds and it now broken. Presumably something somewhere got updated

I'd start investigating why it works differently with vite serve and vite build. Since Vite only serves the modules in development, it could be a good start to understand what is happening in the browser.

We have two apps, one that experiences this problem, the other doesn't and works seamlessly.

Could you share a minimal repro of your apps as well @lxsmnsyc? I think it would help.

[RobWalker]

Try out this example:
https://stackblitz.com/edit/vitejs-vite-uvhc2g

If you run just vite in the terminal and click the button everything works fine, the Uppy dialog opens

If you run vite build then vite preview then it builds OK but you get a runtime 'Buffer' is not defined error on startup:

[RobWalker]

This example has the buffer package used as a polyfill, but runs into the Writable error: https://stackblitz.com/edit/vitejs-vite-7zeshv

[thiagomajesk]

Thanks for the repros @RobWalker. I think this thread has a lot of useful information already.

@Murderlon do you have any sort of ETA for paid customers? I have an open support ticket about this issue. I'm migrating a project to modern frontend tools and this open issue is the only thing preventing us to go forward with our plan. Since Transloadit is a big part of our application, we currently don't have any course of action to pursue while this is still pending.

[RobWalker]

@thiagomajesk my current work around is to load uppy from the CDN rather than bundle it via vite, basically add:

<script src="https://releases.transloadit.com/uppy/v2.2.3/uppy.min.js"></script>

to index.html and then use window.Uppy to reference it from the code ... it is far from ideal, but it does work and our project still has a ways to go before hitting production and we can punt on this for a little longer in the hope of a more formal fix.

[thiagomajesk]

@RobWalker I've been considering this, but as you said is not completely ideal for production. Even more so if you are working on applications that can be served in a standalone fashion (eg: on local networks with limited web access and so on). I'll try this while we wait for an official solution 🙃.

[Murderlon]

Thanks for the updates. I have been trying to get this to work today but it has been less obvious than I hoped to solve it. I understand this is a timely issue and my goal is to get it released this workweek.

[lxsmnsyc]

@thiagomajesk I am afraid that I can't repro this. It's confusing since two projects that shares the same code has difference in success. I'll probably check the CDN work around if that ever solves it temporarily. At least, just to keep everything consistent.

[Murderlon]

It was a bit of a confusing process to reproduce this with only socket.io-client and not through @uppy/transloadit but I did in the end. I reported it to them (see reference above) and offered interest in contributing, but it seems I can't fix this on our side as it seems to be a dual publishing issue on their end.

[Murderlon]

This now has been fixed! It was fixed in engine.io-client, which through semver is automatically in socket.io-client, and in turn in @uppy/transloadit.

Here is a minimal working example: https://stackblitz.com/edit/vitejs-vite-vgvw5x?file=main.js

Reinstall @uppy/transloadit to make it work.

[thiagomajesk]

Thanks a lot @Murderlon! I'll give it a go as soon as I can. Cheers!