XAR lets you package many files into a single self-contained executable file. This makes it easy to distribute and install.
A .xar file is a read-only file system image which, when mounted, looks like
a regular directory to user-space programs. This requires a one-time
installation of a driver for this file system
(SquashFS).
XAR is pronounced like "czar" (/t͡ʂar/). The 'X' in XAR is meant to be a placeholder for all other letters as at Facebook this format was originally designed to replace ZIP-based PAR (Python archives), JSAR (JavaScript archives), LAR (Lua archives), and so on.
There are two primary use cases for XAR files. The first is simply collecting a number of files for automatic, atomic mounting somewhere on the filesystem. Using a XAR file vastly shrinks the on-disk size of the data it holds. Compressing to below 20% of the original size is not unheard of. This can save multiple gigabytes per machine and reduce random disk IO. This is especially important on machines with flash storage.
The second use case is an extension of the first -- by making the XAR file
executable and using the xarexec helper, a XAR becomes a self-contained
package of executable code and its data. A popular example is Python
application archives that include all Python source code files, as well as
native shared libraries, configuration files, other data.
This can replace virtualenvs and PEX files with a system that is faster, has less overhead, is more compatible, and achieves better compression. The downside is that it requires a setuid helper to perform the mounting.
-
SquashFS looks like regular files on disk to Python. This lets it use regular imports which are better supported by CPython.
-
SquashFS looks like regular files to your application, too. You don't need to use
pkg_resourcesor other tricks to access data files in your package. -
SquashFS with Zstandard compression saves disk space, also compared to a ZIP file.
-
SquashFS doesn't require unpacking of
.sofiles to a temporary location like ZIP files do. -
SquashFS is faster to start up than unpacking a ZIP file. You only need to mount the file system once. Subsequent calls to your application will reuse the existing mount.
-
SquashFS only decompresses the pages that are used by the application, and decompressed pages are cached in the page cache.
-
SquashFS is read-only so the integrity of your application is guaranteed compared to using virtualenvs or unpacking to a temporary directory.
Optimizing performance (both space and execution time) was a key design goal for XARs. We ran benchmark tests with open source tools to compare PEX, XAR, and native installs on the following metrics:
- Size: file size, in bytes, of the executable
- Cold start time: time taken when we have nothing mounted or extracted
- Hot start time: time taken when we have extracted cache or mounted XAR squashfs
The PEXs are built with python3 setup.py bdist_pex --bdist-all, and the XARs
are built with python3 setup.py bdist_xar --xar-compression-algorithm=zstd.
| Console script | Size | Cold start time | Hot start time |
|---|---|---|---|
| django-admin (native) | 22851072 B | - | 0.220 s |
| django-admin.pex | 8529089 B | 1.705 s | 0.772 s |
| django-admin.xar | 5464064 B (-36%) | 0.141 s (-92%) | 0.122 s (-84%) |
| black (native) | 1020928 B | - | 0.245 s |
| black.pex | 677550 B | 0.737 s | 0.619 s |
| black.xar | 307200 B (-55%) | 0.245 s (-67%) | 0.219 s (-65%) |
| jupyter (native) | 64197120 B | - | 0.399 s |
| jupyter.pex | 17315669 B | 2.152 s | 1.046 s |
| jupyter.xar | 17530880 B (+1%) | 0.213 s (-90%) | 0.181 s (-83%) |
The results show that both file size (with zstd compression) and start times improve with XARs. This is an improvement when shipping to large number of servers, especially with short-running executables, such as small data collection scripts on web servers or interactive command line tools.
XAR requires:
- Linux or macOS
- Python >= 2.7.11 & >= 3.5
- squashfs-tools to build XARs
- squashfuse >= 0.1.102 with
squashfuse_llto run XARs
This is a setuptools plugin that lets you package your Python application
as a .xar file. It requires squashfs-tools. Install it from PyPI to get
the stable version:
pip install xar
or you can install it from this repository:
python setup.py install
After installation go to your favorite Python project with a console script and run:
python setup.py bdist_xar
The setuptools extension bdist_xar has options to configure the XAR, most
importantly --interpreter sets the Python interpreter used. Run
python setup.py bdist_xar --help for a full list of options.
This is a binary written in C++ used to mount a SquashFS image.
It requires squashfuse installed. Note that the current squashfuse package
on Ubuntu doesn't include squashfuse_ll, so you will have to install from
source.
You can build this part of the code with:
mkdir build && cd build && cmake .. && make && [sudo] make install
Simply run:
python /path/to/black/setup.py bdist_xar [--xar-compression-algorithm=zstd]
/path/to/black/dist/black.xar --help
XAR provides a simple CLI to create XARs from Python executables or directories. We can create a XAR from an existing Python executable zip file, like a PEX.
make_xar --python black.pex --output black.xar
You can also create a XAR from a directory, and tell XAR which executable to run once it starts.
> mkdir myxar
> echo -n "#\!/bin/sh\nshift\necho \$@" > myxar/echo
> chmod +x myxar/echo
> make_xar --raw myxar --raw-executable echo --output echo
> ./echo hello world
hello world
xarexec_fuse will execute the executable it is given using the XAR path as the
first argument, and will forward the XARs arguments after.
First you need to install docker (and possible docker-machine), as it is how it
runs the the code. Then you need to
install the circleci cli, and run
circleci build
If you change .circleci/config.yml you should validate it before committing
circleci config validate
See the CONTRIBUTING file for how to help out.
XAR is BSD-licensed.
