Reduce the initial download size

[jtpio]

Problem

On the demo website the xeus-python files are a bit over 10MB in size:

This size can grow quite significantly as more packages are added by default. For example:

Proposed Solution

Not sure how feasible it is, but maybe there is a way to trim that down a little bit.

Additional context

Also raised by @jasongrout in jupyter-widgets/ipywidgets#3491 (comment).

[jasongrout]

Even if the data was somehow chunked, so only the packages that were actually imported were downloaded, would be great.

[lesteve]

As mentioned above, python_data.data size goes up quickly when adding more packages. For example the command below creates a python_data.data file which is 117MB big which only a few "reasonably standard" packages (scikit-learn,matplotlib-base,seaborn,plotly):

jupyter lite build --XeusPythonEnv.packages=scikit-learn,matplotlib-base,seaborn,plotly

In practice it means that you can not use github pages to deploy your site, since there is a 100MB per-file limit.

I am aware you can use alternative options (e.g. Netlify) but I would guess that pushing into github pages is the simple setup and is likely to be used by a significant portion of people.

Also, for set-ups that push into a gh-pages branch (or .github.io repo) on each commit to main (scikit-learn does that but I am guessing others do as well), even if we are below 100MB, it also means the github repo size may grow quickly, which makes it a bit painful to interact with (e.g. long git commands).

[DerThorsten]

I already implemented such chunking, this will be added in one of the next releases

[lesteve]

Great to hear that, just curious (and to get a better idea of the landscape), in one of the next releases of which project? jupyterlite, emscripten-forge, jupyterlite-xeus-python, something else?

[martinRenou]

This was done in empack, the tool that we use for "packing" the Python environment, we still need to make use of it in jupyterlite-xeus-python.

[lesteve]

Nice, thanks!