Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making 'three' tree-shakeable #24199

Closed
pschroen opened this issue Jun 7, 2022 · 122 comments
Closed

Making 'three' tree-shakeable #24199

pschroen opened this issue Jun 7, 2022 · 122 comments

Comments

@pschroen
Copy link
Contributor

pschroen commented Jun 7, 2022

The goal with tree-shaking here is I should be able to import a class from 'three', and only get the classes needed. In my test bundle importing only Vector2 is producing a 295 KB (uncompressed) file still with lots of remaining side-effects even after r141 and all the work done on #24006 (down from a 1 MB bundle in r140).

I'm opening this issue to resolve the remaining side-effects, which is do-able with some more work, and we have a couple ways of testing that.

Also to make the claim that 'three' is finally fully tree-shakeable, we'll need to verify that in the most popular bundlers. I'll start with Rollup, Webpack, Parcel and esbuild. Open to suggestions of other bundlers and configurations, and will start with a simple test of importing only Vector2.

Steps to reproduce the behavior:

import { Vector2 } from 'three';

console.log(new Vector2(0, 1));

And with agadoo:

npx agadoo
...
Failed to tree-shake build/three.module.js

It's worth noting that importing from the source files with agadoo also fails, and is something I can look into as well.

npx agadoo src/Three.js
...
Failed to tree-shake src/Three.js

The expected behavior, regardless of agadoo, is simply looking at the output bundle. If I import Vector2, I'm expecting only the @license header comment, and Vector2, nothing else.

References:

@marcofugaro
Copy link
Contributor

Other related discussions: #21667 #21665

@pschroen
Copy link
Contributor Author

pschroen commented Jun 7, 2022

@mrdoob @Mugen87 @donmccurdy @LeviPesin I'd also like to use this issue to discuss the use of instanceof, does anyone have an example of it not working?

My fear here is that something has been misunderstood with how tree-shaking works and has led us all on a wild goose chase. Would be great to get some insight from either @Rich-Harris or @lukastaegert on this? See #24006 (comment).

In my experience it's always been working for me, in Rollup at-least (and I've been using it since 2016). For example, I've been using my own TextureLoader, and when bundling my own classes in with other libraries the bundler should be renaming conflicting object names so when they are instantiated instanceof will always be comparing the correct constructor class:

  • TextureLoader becomes TextureLoader$1.
  • So the actual code output in the bundle would be loader instanceof TextureLoader$1, which works.
  • Code minification also works so long as the object names all match.

If there is an issue here, in my opinion that would actually be an issue with the bundler or minification, not with the use of instanceof.

Related discussion: #24167

@LeviPesin
Copy link
Contributor

LeviPesin commented Jun 7, 2022

Does instanceof work with tree-shaking?
I.e. if, for example, WebGLRenderer has .isPointsMaterial check in its body but does not have PointsMaterial import, does the changing it to instanceof PointsMaterial and importing PointsMaterial will break tree-shaking or not? I.e. will the PointsMaterial class be included?
Related: #24006 (comment)

@pschroen
Copy link
Contributor Author

pschroen commented Jun 7, 2022

Oh I think I get why this might be an issue, is the problem simply that PointsMaterial would be included in the bundle?

Technically it is being used in that case, and would be included in the bundle. If the goal is to not include any of the materials with WebGLRenderer, then ya we would have to use .isPointsMaterial.

I'll add this to my list of things to investigate...

@LeviPesin
Copy link
Contributor

LeviPesin commented Jun 7, 2022

Oh I think I get why this might be an issue, is the problem simply that PointsMaterial would be included in the bundle?

Yes.

If the goal is to not include any of the materials with WebGLRenderer, then ya we would have to use .isPointsMaterial.

I think we should calculate how often such pattern is used (i.e. the class is not imported, but .isClass is used)...

@marcofugaro
Copy link
Contributor

I.e. if, for example, WebGLRenderer has .isPointsMaterial check in its body but does not have PointsMaterial import, does the changing it to instanceof PointsMaterial and importing PointsMaterial will break tree-shaking or not? I.e. will the PointsMaterial class be included?

Yup, it will be included. This is the reason three switched to .is* in #9310.

@LeviPesin
Copy link
Contributor

And there is a more detailed explanation in #9310 (comment).

BTW (completely unrelated to this issue but related to that), is the reasons for not using default exports valid now? They are very useful to avoid constantly writing import { Abc } and export { Abc } (they are already used in e.g. the nodes system).

@marcofugaro
Copy link
Contributor

BTW (completely unrelated to this issue but related to that), is the reasons for not using default exports valid now? They are very useful to avoid constantly writing import { Abc } and export { Abc } (they are already used in e.g. the nodes system).

Code style I believe.

@LeviPesin
Copy link
Contributor

And there is a more detailed explanation in #9310 (comment).

In that comment the following link was given:
#4776 (comment)
(It is said there that polymorphism can be used instead of .is checks)
I think we can use the same pattern in most such occurences?

@pschroen
Copy link
Contributor Author

pschroen commented Jun 7, 2022

is the reasons for not using default exports valid now?

Code style I believe.

Agree with @marcofugaro, it's an opinionated topic as a result.

Using named exports is supposed to be friendlier for tree-shaking when you are always explicitly using the named imports, but in practice I've found it makes no difference in my experience, the bundler renames all the objects anyways, and I use both default exports and named exports, though over the years I've moved more towards named exports/imports for consistency and being more explicit.

@LeviPesin
Copy link
Contributor

I think we can use the same pattern in most such occurences?

E.g. the first occurence of .is that GitHub search shows - https://github.com/mrdoob/three.js/blob/dev/src/renderers/shaders/UniformsUtils.js - can be easily fixed by changing the value.isVector2 || ... to value.clone !== undefined. Will file a quick PR for this.

@donmccurdy
Copy link
Collaborator

Sorry I don't see how changing obj.isPointsMaterial to obj instanceof PointsMaterial could ever benefit tree-shaking. Using instanceof will certainly create a reference to the class and prevent tree-shaking, and .is does not...

@LeviPesin
Copy link
Contributor

LeviPesin commented Jun 7, 2022

let's not remove .is properties generally please. See #24199 (comment)

My suggestion is not to replace .is with not-tree-shakable instanceof - my suggestion is to replace it with polymorphism and duck typing, like in #4776 (comment) and #24202.

(I should note that I am not talking about duck typing classes via unrelated methods, I am talking about duck typing them with only the methods used in the code block - like in #24202)

@pschroen
Copy link
Contributor Author

pschroen commented Jun 7, 2022

@donmccurdy Yes agreed, there's been some confusion in the terminology, when I read that instanceof doesn't work with tree-shaking I was taking it literally, technically it does work, it just works against this library because of the way the types are being checked.

I think there's some good discussion here on alternative approaches as above. I'm going to focus more on removing the remaining side-effects in the meantime.

@donmccurdy
Copy link
Collaborator

My suggestion is not to replace .is with not-tree-shakable instanceof - my suggestion is to replace it with polymorphism and duck typing

But why? I guess I don't see any benefit to this — just a lot of work, harder-to-read code, and a hard-to-predict chance of causing classic performance pitfalls associated with polymorphism. If there is some benefit, let's discuss it in another thread since it doesn't seem to be related to this one?

@pschroen
Copy link
Contributor Author

pschroen commented Jun 7, 2022

Agreed, even though related, the bundle being tree-shakeable in its current state doesn't hinge on the type check approach. 👍

@pschroen
Copy link
Contributor Author

pschroen commented Jun 8, 2022

Alright, so I've spent some more time tonight with the latest, here's a summary of my findings. 😉

  • src/Three.js itself has side-effects at the bottom because of the checks for __THREE_DEVTOOLS__ and multiple instances. Tree-shaking: Move registration event and multiple instances check #24225
  • Three.Legacy.js is fully tree-shakable now, nice work guys! 🎉 Tree-shaking broken by Object.defineProperties( ... ) #24006
  • The DataUtils.js file has a number of expressions and for loops causing side-effects. Tree-shaking: DataUtils #24218
  • The PropertyBinding.js, KeyframeTrack.js, Materials.js, Material.js, Object3D.js, Euler.js, Texture.js and Color.js files have static and prototype properties that could be converted to class fields. Have we made a decision on the use of class fields?
  • Material.js has a side-effect at the bottom with Material.fromType and a // TODO: Behavior added in Materials.js, can that be removed? Core: allow tree-shaking again for materials #24094
  • ShaderLib.js, UniformsLib.js and ShaderChunk.js are also side-effects, perhaps there's a different approach we could take with defining these shaders? Tree-shaking: ShaderLib and UniformsLib #24221
  • ObjectLoader.js creates a big side-effect with the Geometries namespace, removing that one line allows for tree-shaking.
  • WebGLRenderer.js of-course is loading a lot of classes including the shader classes with side-effects, which in-turn creates a domino effect importing all the classes from ShaderLib.js and UniformsLib.js, etc., removing those classes allows for tree-shaking as well.
  • And finally MathUtils.js has a side-effect at the top with the _lut for loop. Tree-shaking: MathUtils #24210

Removing all of the above, shader and renderer classes from src/Three.js is allowing me to build a module bundle with npm run build-module and then in-turn import from build/three.module.js side-effect free.

npx agadoo

Success! build/three.module.js is fully tree-shakeable

Still lots of work to do but we're getting there! 😅

@lukastaegert
Copy link

So while relying on duck typing alone is probably best for tree-shaking, instanceof is an improvement as it allows for some basic static analysis. I created rollup/rollup#4524 for you, you can try out if this would help.

@mrdoob
Copy link
Owner

mrdoob commented Jun 8, 2022

@lukastaegert Nice!

Maybe we can try using instanceof with Vector2, Vector3 and Vector4. That should solve #24167.

@pschroen Would you like to give it a go?

@pschroen
Copy link
Contributor Author

pschroen commented Jun 8, 2022

Sure, though I feel like it might be a bigger can of worms for consistency? I am a fan of a hybrid approach for type checks, so maybe .is properties for classes used by WebGLRenderer to keep the bundle size down, and then instanceof for the math classes? Is there any performance impact with instanceof compared to .is?

For reference I was looking at this file first.

} else if ( value?.isVector2 === true ) {

@mrdoob
Copy link
Owner

mrdoob commented Jun 8, 2022

Is there any performance impact with instanceof compared to .is?

I want to think that instanceof is faster... 🤞

@LeviPesin
Copy link
Contributor

So now (after merging that PR, actually) when Rollup tree-shakes instanceof, we can convert all of the .is checks to instanceof?

@marcofugaro
Copy link
Contributor

when Rollup tree-shakes instanceof, we can convert all of the .is checks to instanceof?

No, this would make the whole repo un-treeshakeable again 😅

Rollup isn't the only bundler out there.

@LeviPesin
Copy link
Contributor

LeviPesin commented Jun 8, 2022

I think converting some .is* to instanceof and not converting others would only create a mess.

Maybe we should create an issue for tree-shaking instanceof in other popular bundlers repositories? Or even a PR based on rollup/rollup#4524.

@marcofugaro
Copy link
Contributor

marcofugaro commented Jun 8, 2022

Maybe we should create an issue for tree-shaking instanceof in other popular bundlers repositories? Or even a PR based on rollup/rollup#4524.

Good luck with that, we tried in the past with tree-shaking of .prototype. and it was never supported.

@LeviPesin
Copy link
Contributor

ES6 classes are just cleaner and easier to read.

@pschroen
Copy link
Contributor Author

Hey @marcofugaro @Mugen87 @LeviPesin 👋,

Ok, what animation classes are you talking about?

All the src/animation/** classes really, specifically the PropertyBinding and KeyframeTrack classes are all using .prototype.* properties, this is the normal way of writing these modules though and I don't think it's worth refactoring. For these particular problems I think it's more a problem of modernization, so either webpack needs to add tree-shaking support for .prototype.* properties, or we move to class fields, whichever comes first. 😅

I think private fields? See discussion in #21235

When there is no other reason than improving the code style, I would not vote to touch the renderer's code base.

ES6 classes are just cleaner and easier to read.

Hmm, I see, ya @Mugen87 you're right it's primarily code style and syntactic sugar, though there is one small case with WebGL1Renderer that has a .prototype.* property and is included in the bundle regardless.

WebGL1Renderer.prototype.isWebGL1Renderer = true;

This is a can of worms question, are there any other ways of defining a WebGL 1 rendering context, perhaps by passing-in a parameter to the constructor?

@Mugen87
Copy link
Collaborator

Mugen87 commented Jul 12, 2022

This is a can of worms question, are there any other ways of defining a WebGL 1 rendering context, perhaps by passing-in a parameter to the constructor?

You can pass in a custom rendering context to WebGLRenderer.

@pschroen
Copy link
Contributor Author

Ya I'm not sure if deprecating the WebGL1Renderer class is worth the 2 lines of code that gets bundled. 😆

@Mugen87
Copy link
Collaborator

Mugen87 commented Jul 12, 2022

No, I don't think so^^.

@pschroen
Copy link
Contributor Author

As an update on tree-shaking the shaders, I've spent some time yesterday and today trying to make them fully tree-shakeable, I've almost got it working, but hit a couple blockers preventing me from finishing the work.

  1. It's especially difficult making three.js tree-shakeable with the flat bundle, and almost contradictory exporting all the shaders so they can be overridden, while at the same time tree-shaking them all when not used.

  2. When converting the contents of ShaderLib and UniformsLib to either named exports, or even individual files, Rollup always bundles the namespaces in the flat bundle as frozen objects with Object.freeze(), so even though I can get this working with tree-shaking it then breaks some examples and RectAreaLight because of the way we're using them as writable objects for custom shaders, for example:

ShaderLib[ 'line' ] = {

UniformsLib.LTC_FLOAT_1 = new DataTexture( ltc_float_1, 64, 64, RGBAFormat, FloatType, UVMapping, ClampToEdgeWrapping, ClampToEdgeWrapping, LinearFilter, NearestFilter, 1 );

  1. Again this all works when importing from the source files, these are blockers on the flat bundle only. I realize we're only supporting the flat bundle, but if the goal here is to allow people to modify them, wouldn't it make more sense to be working with the source files then?

  2. Regardless, I think anyway we approach it we'll need to get rid of the ShaderChunk, ShaderLib and UniformsLib namespaces. I have this working locally with the shaders all self-contained in each of the Materials, and it's working great, the main blocker for me right now is supporting the "custom" shader use cases.

  3. It's worth noting extending a material works as a way to override the shaders, I'm specifically talking about custom shaders that are built using ShaderChunk, ShaderLib and UniformsLib directly.

How would you guys feel about possibly opening-up a use case where the src/* files aren't just there for "rhetoric", but as a possible use case for custom shaders? Like instead of importing from 'three' for a particular shader by name, you could just import from the source file? This way we could effectively remove all these namespaces from the flat bundle... 🤔

@pschroen
Copy link
Contributor Author

Ya, @marcofugaro I've hit the same dead-end with #include <...> and ShaderChunk. I was able to fully remove ShaderLib and UniformsLib, but the entire shader system of three.js is built around these includes.

And the savings from ShaderLib and UniformsLib alone are not enough to justify it. So super long story short, if we go any further it would be a huge breaking change now at this point. This library is really built to function around having all the shaders available in the flat bundle, and it also allows for custom shaders from the flat bundle if we leave it like that.

Well, I gave it my best shot guys, and it's been a good learning experience being knee-deep in the three.js renderer and shaders. There's not much we can do about the remaining static and .prototype.* properties either:

I think it's more a problem of modernization, so either webpack needs to add tree-shaking support for .prototype.* properties, or we move to class fields, whichever comes first. 😅

After #24336 is merged I think we can close this issue now, we've gone as far as we can with the shaders and webpack for now. 🫠

@donmccurdy
Copy link
Collaborator

donmccurdy commented Jul 15, 2022

Is there a summary of the issue with #include <...>? I can see it is a problem that the #include macros refer to a shared global namespace. Perhaps that could be replaced with namespaces specific to a material type?

import { x } from 'path/to/chunks/x.glsl.js';
import { y } from 'path/to/chunks/y.glsl.js';

class MeshStandardMaterial extends Material { ... }

MeshStandardMaterial.ShaderChunk = { x, y };

export { MeshStandardMaterial };

Still a breaking change, but manageable. Or have I missed the issue?


Another consideration is that the node-based material system is still in development, and will be the primary system for WebGPU, I assume. I would guess its interaction with tree-shaking is quite different, and probably more flexible.

@pschroen
Copy link
Contributor Author

You got it, that would be the next step I was going to take but feeling it's not really worth it. See #21665 (comment)

Writing custom shaders with three.js is already a little awkward, and I think this would make it even more complicated. It's more convenient having all the chunks in one place I think. 🙃

@gkjohnson
Copy link
Collaborator

I can see it is a problem that the #include macros refer to a shared global namespace. Perhaps that could be replaced with namespaces specific to a material type?

My impression is that there are a few issues that could ultimately affect the end user if not handled properly. Take this all with a grain of salt but this is what I've gathered over from roughly following these conversations:

  1. The main shader materials all rely on these shader chunk includes which reference a map of shader chunks by string id which aren't tree shakeable.
  2. These could be removed in a variety of ways (inlining using templates, for example, removing #includes entirely) but ultimately end users have a lot of code that rely on pattern matching those include statements to augment built in shaders or use them in their own shader code.

Point 2 is what makes this hard because it means we can't just remove the #include statements outright or only just include the ones required for just the instantiated built-in materials. Not to add more noise to all this but there may be some approach with a minimal migration path for existing users that implicitly rely on some of these shader chunks.


  • Add a registration-style "ShaderChunkLib" that affords registering shader chunks and uniform sets.
  • In the constructor of all the built-in materials each required shader chunk and uniform set could could be registered.
  • Then users could register their own shader chunks / uniform sets (including built-in ones that aren't implicitly registered).
  • A function like registerAllChunks could be included that existing users could run to enable all their existing shader code to continue to work until they've move to import just what's needed.

If I'm understanding all this tree shaking stuff then that should afford the ability for only the required shader chunks and uniform sets to be included in a tree-shaken bundle. The built-in material code might look like this:

import { bsdfs, uv_pars_fragment, ... } from 'path/to/chunks/ShaderChunk.js';
import { ShaderChunkLib } from 'path/to/chunks/ShaderChunkLib.js';

class MeshStandardMaterial extends Material { 

  constructor() {

    super();
    ShaderChunkLib.register( 'bsdfs', bsdfs );
    ShaderChunkLib.register( 'uv_pars_fragment', uv_pars_fragment );
    // ...

  }

}

// ... migration function for end users

import { registerAllShaderChunks } from 'three';

registerAllShaderChunks();

// all built-in chunks available for user-written shaders

Admittedly the above also looks like a huge pain to maintain. But perhaps there's a more svelte pattern for this (like what @donmccurdy suggested). Though at the least it seems like something like this would get things working without dramatically impacting end users.

@pschroen
Copy link
Contributor Author

Interesting, ya something that's evaluated at runtime would work much better with tree-shaking, to quote agadoo again:

Don't create instances of things on initial evaluation — instantiate lazily, when the functions you export are called

@LeviPesin
Copy link
Contributor

One more related discussion:
#23368 (comment)

@pschroen
Copy link
Contributor Author

pschroen commented Dec 3, 2022

As an update on agadoo, Rich Harris has updated it to the latest version of Rollup with the release of v3.0.0 (Rich-Harris/agadoo#16), and as a quick unit test:

git clone --depth=1 https://github.com/mrdoob/three.js.git
cd three.js
npx agadoo

@looeee
Copy link
Collaborator

looeee commented Dec 4, 2022

npx agadoo fails, however if I remove these lines:

if (typeof __THREE_DEVTOOLS__ !== "undefined") {
	__THREE_DEVTOOLS__.dispatchEvent(
		new CustomEvent("register", {
			detail: {
				revision: REVISION,
			},
		})
	);
}

if (typeof window !== "undefined") {
	if (window.__THREE__) {
		console.warn("WARNING: Multiple instances of Three.js being imported.");
	} else {
		window.__THREE__ = REVISION;
	}
}

Then run npm build and npx agadoo again, it succeeds. So I guess these globals are the last remaining issue.

@Mugen87
Copy link
Collaborator

Mugen87 commented Dec 4, 2022

It seems the three.js dev tools are not maintained for quite a while anymore so we can probably remove the __THREE_DEVTOOLS__ related code.

The multiple instance check is helpful in certain cases but I wonder if it's more important than full tree shake support. Granted, it's a bit of a hack that the library writes to window...

@donmccurdy
Copy link
Collaborator

donmccurdy commented Dec 4, 2022

The multiple instance check is helpful in certain cases but I wonder if it's more important than full tree shake support.

Personally I think the multi-import check is more helpful, yes, if everything except the 6-7 lines of code required are being tree-shaken. If dev tools were actively maintained I would vote to keep those lines too, but as it is... perhaps not. The difference between 99.9% tree-shaking and 100% tree-shaking is basically nothing.

Or it may be possible to wrap those lines in a block like this:

if (process?.env?.NODE_ENV === 'development') {
  // ...
}

Not sure about agadoo, but bundlers should strip that code except during local development.

@marcofugaro
Copy link
Contributor

Or it may be possible to wrap those lines in a block like this:

Nope, bundlers do a string replacement of the process.env.NODE_ENV word with the string "development" (or "production") at build time, so the example you posted doesn't work.

The multi-import check has been proven useful for me multiple times, so I vote we leave it.

@donmccurdy
Copy link
Collaborator

Probably possible to work around that?

const process = typeof global === 'undefined' ? {env: {}} : global.process;

if (process.env.NODE_ENV === 'development') {
  // ...
}

But either way, I don't think having those few lines remaining is a problem.

@looeee
Copy link
Collaborator

looeee commented Dec 4, 2022

if everything except the 6-7 lines of code required are being tree-shaken

This is a good question, which of these is true?

  1. The rest of the library is correctly tree-shaken and just these lines remain
  2. Bundlers encounter these lines and bail out of tree-shaking entirely

If it's 1. then I don't see any problem with these lines remaining.

@marcofugaro
Copy link
Contributor

marcofugaro commented Dec 5, 2022

  1. Bundlers encounter these lines and bail out of tree-shaking entirely

In my experience, that doesn't happen.

@mrdoob
Copy link
Owner

mrdoob commented Dec 6, 2022

Should we close this then?

@Mugen87
Copy link
Collaborator

Mugen87 commented Dec 6, 2022

I would say yes. Related topics like the usage of class properties can be discussed in specific PRs or issues.

@pschroen
Copy link
Contributor Author

pschroen commented Dec 6, 2022

Agreed, you want to do the honours, or should I close it? 😅

@mrdoob
Copy link
Owner

mrdoob commented Dec 6, 2022

🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests