Update either the compiler or runtime to allow independently built components to share runtime functions

[ghost]

Edit: A proof of concept implementation can be found in this comment: #3671 (comment)

Is your feature request related to a problem? Please describe.
The lack of this feature leads to bugs in stand-alone pre-built components which are being imported into another Svelte code base.

Bugs are always related to runtime internals because a stand-alone component is always built with its own version of the internals.

For example, it's impossible to use the context API in a pre-built stand-alone component when it has been included in another Svelte code base. Another example would be bind:this.

I understand that the "svelte" key in package.json exists to allow you to import components from other projects at build time and have them packaged up to use the same internals.

Unfortunately there are some applications that can't know what components they will import until runtime and can't build and package up any old Svelte component they find on the fly to use the same internals, so this is a separate issue.

Describe the solution you'd like
Edit: An easier to implement solution based on just altering runtime exports is described in my next comment in this issue.

First of all, I understand this isn't a trivial change, but that being said, I feel like this use case is common enough that it would be worth adding an option to the compiler to compile a component without any internals and for the component to expect to be given an internals object, or something of that kind, which it can then use in place of having its own copy of the internals.

In other words, I'd like to have a compiler option like plugin: true, or something like that, which signals that this component must be used within a project that has some internals to pass on to it.

I haven't looked at all of the code in the runtime, but I don't see why this shouldn't be possible, since most of the internals that cause the problems are things like module level array or object references that get imported into various files to push functions etc. onto.

If as an architectural decision Svelte is ok with module level references being imported all over the place, then I don't see why we shouldn't just allow a component to take a reference to those same objects if they are optionally passed into a component.

The main problem I see with this is that as Svelte versions move on the "plugin" component and the main code base may drift apart to the point where they are no longer compatible, but this seems like a detail that application developers should be keeping in mind. I think it's ok for Svelte to simply provide the option to pass internals on and after that's happened it's the application developer's responsibility to keep things in sync.

I can also see there being a problem with dead code removal in the final build of the main application, since it would have to pass through all of the internals.
Given that this is useful for applications that don't know what components they'll be using as runtime that's just one of the trade offs, but perhaps that could just be an option as well?

Describe alternatives you've considered
I've considered just avoiding the issues by not using features like bind:this or the context API in "plugin" components, but I can't shake the feeling that this should just work, given that I can't see any good reason it shouldn't (admittedly in the limited time I've spent looking at the Svelte code base).

Other than that I'm not sure of other ways that internals could actually be shared.
I understand that this could be a change that makes the code base too complex, and if that's the case I'm fine just not using the features that aren't supported, but I wanted to at least throw the idea out there.

How important is this feature to you?
It's extremely important to me, as someone who builds applications that can't know in advance what components it will be using.

I can work around it by just not using those features, but it's frustrating knowing that I could use those features if Svelte would just (optionally) share its internals a little more.

[ghost]

Edit: A proof of concept implementation can be found in this comment: #3671 (comment)

Another approach to this that would be much simpler to implement would be for the functions in the runtime (because that's where all of these problems seem to originate) to define their function on a svelte object on either window or global on their first import.

After that the main problem is the current_component imports, which happen in two files: src\runtime\internal\ssr.ts, and src\runtime\internal\Component.ts.

I'm guessing this is purely to avoid the error that's thrown in get_current_component when no parent component exists yet. In that case it could just take an optional argument; the parameter could be something like options = { skip_error_check: false } and could be used to skip the check in those two files since they don't do any error checking anyway. That way we always call the function to return the correct variable even when there are multiple versions of the runtime internals.

Those runtime files could then export either the existing global.svelte.* functions or define them and then export so all components always use the same internals, which will always be the first ones loaded in the main host project.

I realise that this probably isn't the solution some will want because I used the word "global", but perhaps it could be an option that's turned on by a switch/env var/whatever, like SVELTE_GLOBAL_INTERNALS=1, for example, since you'd only use it in projects that need to host independent pre-built components. Doing this would allow projects that want to use global internals to opt-in, so the default would still be the current way which allows for dead code removal through tree shaking, since we can make it conditional based on a constant replaced using the rollup replace plugin.

I also don't see the problem with a global store for runtime functions since the goal of this is for stand-alone components to all use the same functions in the end. It does mean that stand-alone components can't choose to be built without internals, but the issue here isn't file size, (otherwise I'd be building everything in the same project at great expense to flexibility), rather the issue is individual components not using the same runtime functions.

Having said that, there could be an option for "plugin" style components that just define runtime functions as whatever is on global.svelte so the actual definitions can be removed during dead code removal to cut down on size at least a little.

I've altered most of the runtime files in a local build of Svelte and it's working great so far without any obvious problems. It solves literally every problem I've had so far with pre-built standalone components, including a few that there are open issues about, like components erroring on outros.c etc. (though I realise those open issues may not necessarily be about the same problem I'm describing).

There is one error about not being able to tree shake internal\index.mjs when I run the tests, but that's expected since the whole point is to make sure all of the runtime internals are available in the main host project. Other than that all of the actual tests pass.

Could anyone let me know of any downsides to this approach that don't unnecessarily bikeshed on globals or unused Svelte internals (that's a trade off I was making anyway) ?

[paymonsattar]

I would also appreciate this being done. Currently making an application with modular extensions and I have also come across this.

[ghost]

As an advocate for wanting to expose and use global runtime internals between independently built components I've taken it upon myself to at least prove how easily this can be done and compared final file size in a simple project.

You can see the branch in my fork here: https://github.com/jakelucas/svelte/tree/optional-global-runtime-internals (no longer available)

And you can see the comparison to the master branch I forked here: https://github.com/jakelucas/svelte/compare/master...jakelucas:optional-global-runtime-internals (no longer available)

First of all, some rough measurements in a basic project.

Using the current version of plain Svelte my index.js file was 74.4KB uncompressed (Brotli 26.9KB), so I used that as a baseline.

With my changes for optional global runtime internals turned off my project's index.js file became 74.6KB uncompressed (Brotli 27.0KB).

With my changes for optional global runtime internals turned on my project's index.js file became 84.5KB uncompressed (Brotli 30.9KB).

The changes I've made make use of the rollup replace plugin to replace '__GLOBAL_INTERNALS__' with a constant boolean based on a SVELTE_GLOBAL_INTERNALS environment variable. This allows the bundler to properly use tree shaking to remove the global internal runtime code when the environment variable is not set.

This means the global runtime internals are optional and are an opt-in feature so that users who don't care about using independently built components together don't pay the price unnecessarily. This also means that unless a component was built using SVELTE_GLOBAL_INTERNALS=1 or something similar it won't use the global internals even if they exist anyway.

The main changes to the code base are all in the src/runtime/internal folder, since this is where all of the runtime errors originate due to use of module scoped objects and arrays to store things like the current component etc.

By checking for a global function before exporting it's possible to make sure that all independently built components use the same runtime functions and therefore the same module level variables that track state. This fixes all of the problems (that I know of) relating to separate copies of the runtime being used.

The only downside to exporting global functions optionally is that the export code has become a little messier, but since it doesn't affect too many files/functions I don't think this is actually a problem in the grand scheme of things.

The Component.ts file and the ssr.ts files have been altered to replace the direct use of current_component with a call to get_current_component({ skip_error_check: true }) to make sure they always get the correct component from the module level state tracking variable. The new skip_error_check option is there because I assume direct use of the variable was to avoid the error check when a parent component didn't exist yet, so this remains functionally equivalent.

The global object is returned from a new get_global function in the utils.ts file that throws an error if it's run when __GLOBAL_INTERNALS__ is false, and does a basic version check between the current component and the existing version of the runtime being used to display any warnings about incompatibilities in the console.

I don't believe it's up to Svelte to support independently built components working together from different versions and it's up to the application developer to make sure versions between components match up.

All normal unit tests pass when global internals are both turned on or off. The only test that doesn't pass is the agadoo tree shaking test when global internals are turned on, but this is completely expected and is part of the whole point anyway, since a project using global internals would need to make them all available.

Lastly, I don't know much about TypeScript, so I added the following lines to utils.ts to pacify the compiler:

declare const global: any;
declare const window: any;

If there's a better way to do this please let me know.

I've been using this patch in my project and it's fixed all of the problems I've run into without any need for change in the actual compiler.

Since it's opt-in I believe this is a nice compromise to solving the problem as it doesn't affect users unless they explicitly say they want to use it whilst only making the export code in a few files slightly messier; otherwise Svelte functions as it always has.

Using this solution enables applications that don't know what components they will be using until runtime (because the code doesn't exist in the project in the first place, for example) to use pre-built components from the same version of Svelte without having to be recompiled and bundled up again.

It's true that a host project importing externally built components will be wasting some bandwidth on Svelte runtime internals that might never be used in the host project, and will never be used in the external components, but that's a trade off those kinds of projects have already decided to make anyway.

In my changes I may have even been a little overzealous in checking for globals for functions during export. It's possible that not all of the functions I chose to export as global would actually need to be exported in that manner, but I wanted to try for the worst case.

What does everyone think to this solution?

[halfnelson]

Could you achieve the same thing by telling rollup of the component library that svelte is an external dependency?, which would leave all the require('svelte/internal') calls in the component library source to be resolved when the main host project is built

[halfnelson]

The problem with both of these approaches is that the compiled svelte components are dependent on the API shape of svelte internals which is not public or stable. eg a component library built with svelte 3.7 would not work correctly with a host using svelte 3.12

[halfnelson]

I think the main conflicts happen when a parent component assumes a child component is part of the same runtime instance and calls the shortcut methods (eg $$render instead of .render). Maybe what is needed is for the compiler to emit different construction logic for subcomponents from different svelte instances ( although the cost might not be worth the gains ).

[ghost]

I'm not sure at all how to configure rollup so I don't know about the first question. The main point of this though is to not need to rebuild the host project in the first place.

My use case for this is essentially to have a host project that can pick up built JS file components from a folder and use them at runtime without having to recompile with the knowledge of all possible components.

As for depending on the API shape that's why I added the basic version check to emit warnings. That version check is just a proof of concept though and could be strict enough that it requires an exact major and minor version match to not emit a warning.

I don't think it's up to Svelte to support this use case between different Svelte component versions and that's up to the application developer to keep in sync. It's also why I think it's best as an opt-in (in addition to the tree-shaking). The reason for this change is because when independent components use the same version of Svelte I don't see why it shouldn't just work, given that they all use the same functions anyway.

[halfnelson]

"host project that can pick up built JS file components from a folder and use them at runtime" not sure how this would work?, how can you instantiate them from host svelte project without knowing them?

[ghost]

In my particular case the host project just walks through all of the files in a set folder. I build both a X.client.js and X.server.js file for each external component so the correct one can be loaded on both the server and the browser.

All I do is import the file using import() on the server side and <script> tags in the browser.

I then store the returned module in an object with a key based on the file path. Those keys are then looked up at runtime based on data saved in a database and are instantiated in both SSR on the server and in the browser at runtime to generate content.

The reason I want this is so that I can have multiple host projects set up with completely different component sets. This even allows a client to provide their own pre-built components without needing the host project source as well.

[halfnelson]

you might be able to do this without touching the svelte compiler.
something like this in your host startup to share the internals

 import * as SvelteInternals from 'svelte/internals'
 (window || global).__svelte_internals = SvelteInternals

then in the rollup config for your client.js and server.js you would

globals: {
  'svelte/internal': '__svelte_internals'
}

_

[ghost]

Thanks for the suggestion; I had no idea you could do that.

I've tried it in webpack using this in the external component projects:

externals: {
  'svelte/internal': 'svelteInternal',
},

And in the host project I added the suggested code to the entry point. I have two entry points, one for client and one for server, so I used:

import * as svelteInternal from 'svelte/internal'

window.svelteInternal = svelteInternal

and:

import * as svelteInternal from 'svelte/internal'

global.svelteInternal = svelteInternal

It seems to work just fine and is obviously the better and prefered solution. I'm glad I don't have to argue for the runtime to be altered now 👍

I'll leave this issue open for a few days to see if I run into any issues, but if everything works out I'll close it and stick with your suggested solution.

Assuming this solution doesn't bring up any problems perhaps this should be documented?

Thanks!

[ghost]

Well after using the method suggested by @halfnelson it's clear to me that it will work just fine, so I'll close this issue now rather than waiting.

In my case I've taken it to the extreme at the cost of only a couple of kilobytes in the browser.

Basically I've change the webpack externals config to:

externals: /^svelte/

to mark every Svelte related import in my external projects as an external.

Then in the host project I provide everything on window or global like this:

import * as svelte from 'svelte'
import * as svelteAnimate from 'svelte/animate'
import * as svelteEasing from 'svelte/easing'
import * as svelteInternal from 'svelte/internal'
import * as svelteMotion from 'svelte/motion'
import * as svelteStore from 'svelte/store'
import * as svelteTransition from 'svelte/transition'

const g = typeof window !== 'undefined' ? window : global

g.svelte = svelte
g['svelte/animate'] = svelteAnimate
g['svelte/easing'] = svelteEasing
g['svelte/internal'] = svelteInternal
g['svelte/motion'] = svelteMotion
g['svelte/store'] = svelteStore
g['svelte/transition'] = svelteTransition

I'll close this issue now since this is a much nicer solution than trying to change the internals.

[bestguy]

If I'm understanding correctly - is the suggestion that any time an app wants to use 3rd party svelte components the dev would need to manually add the boilerplate above:

import * as svelte from 'svelte'
...
const g = typeof window !== 'undefined' ? window : global
g.svelte = svelte

in addition to the library looking for and assuming these globals?

If so, this still seems pretty fragile and feels like the issue is unresolved and should remain open.

[ghost]

@bestguy

is the suggestion that any time an app wants to use 3rd party svelte components the dev would need to manually add the boilerplate above

No, this isn't the case, and the use case described in this issue isn't a common one (or shouldn't be at least).

There are 3 use cases for components as far as I can see.

For one there should be no problems at all, for another there should be no problems with one caveat, and for the last one (and the caveat) all problems are solved by sharing internals as global (provided you can guarantee all components are built with the same Svelte version).

Svelte project imports another Svelte project (.svelte files)

In this case the library just adds a "svelte" key to their package.json file and the host project bundles everything up as required. Because everything is bundled up in the same project all Svelte components share a single copy of the internals and everything works as it should do.

Non-Svelte project loads pre-compiled JS component files

In this case the host project just loads the pre-compiled JS component file and uses it as a normal JS module. Pre-compiled JS components from Svelte get bundled with their own copy of the internals so everything works out of the box.

There is a caveat to this which I'll get to soon.

Svelte project loads pre-compiled JS component files

In this case the host project is using Svelte, so it's bundled with a copy of the internals, as it should be, but this time, for whatever reason, the library's .svelte files are not being imported and are being loaded via the pre-compiled JS files.

This is fine until the host and the library need to interact.

Because the host has its own internals and the library was understandably compiled with its own internals as well, when it comes time for the host Svelte components to do things like mounting/unmounting the library components communication between them breaks down. This is because there are two copies of the internals that both have their own module level tracking variables that are updated as things happen on the page. If a library sets an onMount function in its own internals then the internals from the host project won't be able to see the function and won't know to run it. This kind of communication breakdown between runtime internal copies also manifests as errors such as outros.c not being defined and things like that.

The solution to this is the one provided above where the host project provides internals as a global and the library just expects them to exist so it can use the same ones. This use case is quite niche since if you have a host Svelte project you would normally just import the library and be done with it, and Svelte projects that need to import pre-compiled components (like my project) are set up in such a way that special logic is required to make everything work anyway, so making globals available isn't exactly a big deal and it's a trade off those kinds of projects just have to make.

Non-Svelte caveat

Now, onto the caveat for the second use case, a non-svelte project using pre-compiled Svelte components as JS files.

If a non-svelte project is using a pre-compiled Svelte library everything will work fine because the Svelte library will have its own runtime internals copy to use. That is, until you try to mix multiple libraries together. If you take Svelte library A and Svelte library B which have both been pre-compiled separately from each other then they will have separate internals. As soon as you try to pass a component from one library into the other there will be a communication breakdown between the internals again and the same errors as a Svelte project loading JS components will appear. This is because it's essentially the same thing even if the main host project isn't Svelte; it's still just one Svelte component trying to use another independently built component with two or more copies of the internals floating around.

There is also of course the more obvious problem of two independently built components using different version of Svelte as well.

As long as library A and B in that scenario never mix then there will never be a problem.

In other words, problems only occur once you start trying to mix two or more copies of the runtime internals.

It looks like you've been directed here by @telemmaite from an issue here: bestguy/sveltestrap#71

They describe their problem here: #3448 (comment)

I tested out SvelteStrap on my machine and I couldn't reproduce the errors they were getting, because as expected you're correctly setting a "svelte" key in your package.json.

But as you can see from this comment (#3448 (comment)) it is possible that even if the main host project is a Svelte one and a library is being imported as recommended, if the main project isn't set up correctly then two or more copies of the internals may still be bundled by mistake, which is the issue I assume they're having.

I thought it may be related to your recommendation to use npm --save rather than npm --save-dev, because there's no need to save as a normal dependency with Svelte since everything is bundled up at compile time, but even using --save in webpack it still worked fine.

Since your pre-compiled version of SvelteStrap seems to be a single JS file where everything is bundled as a single project it means all of your components should be sharing internals as well, so a non-svelte project I assume should have no problem using your library (unless they try to pass other pre-compiled Svelte components into it).

Because of all of this I'm more inclined to believe that if @telemmaite's project is a Svelte one, and they are importing components as they should do, then something is probably wrong with their project set up that it's bundling two or more separate copies of the internals which leads to their errors they described in the issue linked earlier.

Why this issue is closed

I don't think there's a need to reopen this issue because the use case described is quite niche and there's only so much support Svelte would be able to provide for it anyway.

One of the reasons for the way the Svelte project has been written, as far as I understand, is to allow for as much tree-shaking as possible to cut down on bundle size.

Officially supporting the use case described here would mean having to always build with the entire runtime bundled and available as a global all the time, since the host project can't know which functions will be required by any random components being loaded. This would go completely against the grain of the runtime etc. being built in a way where tree-shaking is as easy to provide as possible and would mean users that don't need all of the runtime will have to load it anyway.

Because of that my first suggestion was to be able to turn globals on at build time with an environment variable, but then @halfnelson suggested using the externals or globals keys in webpack/rollup and just attaching to the global object manually instead.

Since a project using components as described by this issue would have to explicitly turn on globals in the first place, the suggestion to just do it manually is much better since it's barely any different as far as setup/boilerplate goes and avoids needing to change the project's internals just to facilitate it.

In addition to that problems will come up if two separately built component's versions differ too much. If one component expects to be able to provide something for another component where the versions differ a lot then it's possible that things will break anyway due to both components possibly not expecting the same behaviour.

Because of all this I can see why it's not an officially supported use case and why the "svelte" key for Svelte projects exists.

[bestguy]

Thanks for the detailed writeup and explanation @jakelucas 👍

[brucejo75]

@jakelucas (whoever you are?),

Thanks for this writeup. Really opened up my options.

Here is the way I handled it:

Create shared Svelte Runtime

So essentially, I wanted to maintain the paths to the svelte internal components. So I put each component in their own file.
And each of the files is very simple, for example:
svelte.js:

export * from 'svelte';

Also, note that I included each file as an entry point so that Vite would output a file for each entry point:

    lib: {
      entry: [
        'src/svelte.js',
        'src/animate.js',
        'src/easing.js',
        'src/internal.js',
        'src/motion.js',
        'src/store.js',
        'src/transition.js'
      ],
    ...
    }

Now my dist folder for my svelte runtime has all the svelte files:

svelte.js
animate.js
easing.js
internal.js
motion.js
store.js
transition.js

Plus, whatever other files the bundler produces that are shared between these files.

Use the svelte runtime

Now in the app/package that uses the svelte runtime:

I am using Vite for my build management.
Instead of using the Rollup globals option. I used the importmap script type. And I thoroughly depend on es modules.

To make sure you do not resolve the imports in your bundler you need to declare them external. The vite.config.js just needs to use the build.rollupOptions.external option, e.g:

external:
  {
    'svelte',
    'svelte/animate',
    'svelte/easing',
    'svelte/internal',
    'svelte/motion',
    'svelte/store',
    'svelte/transition'
  }

Finally, you need your import map.
My importmap look like this:

{ "imports":
  {
    "svelte": "/path/to/svelte/runtime/svelte.js",
    "svelte/animate": "/path/to/svelte/runtime/animate.js",
    "svelte/easing": "/path/to/svelte/runtime/easing.js",
    "svelte/internal": "/path/to/svelte/runtime/internal.js",
    "svelte/motion": "/path/to/svelte/runtime/motion.js",
    "svelte/store": "/path/to/svelte/runtime/store.js",
    "svelte/transition": "/path/to/svelte/runtime/transition.js"
  }
}

Then when you load your package that depends svelte the browser will look up the import map and just load all of the specific files you use in your app/package! Easy peasy.

Happy Svelting 😺