Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

document packaging of environments as self-extracting executables #159

Merged
merged 6 commits into from
Feb 8, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ Use Cases
<https://conda.io/docs/user-guide/tasks/manage-environments.html#sharing-an-environment>`_,
and recreate the environment when needed.

- Packaging an environment as single executable with entrypoint to run on execution (see the
instructions for :doc:`Linux and macOS <unix-binary>`).

- *BETA*: Packaging a conda environment as a standard Cloudera parcel. This is
a newly added capability. It has been tested on a live cluster, but different
cluster configurations may produce different results. We welcome users to
Expand Down Expand Up @@ -179,3 +182,4 @@ This tool has a few caveats.
spark.rst
parcel.rst
squashfs.rst
unix-binary.rst
110 changes: 110 additions & 0 deletions docs/source/unix-binary.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
Bundle as Single Executable
===========================

``conda-pack`` can be used to distribute conda environments as executable shell
scripts for Linux and macOS.


Packaging a Simple Binary
-------------------------

We will package the `normaliz <https://github.com/Normaliz/Normaliz>`_ binary in
this example. It provides a command line tool which is compiled from C++ code.

Create an environment and `conda pack` it:

.. code-block:: bash

$ conda create -y -n normaliz normaliz=3.8.5
$ conda pack -n normaliz

Add an entrypoint that activates the environment and starts normaliz:

.. code-block:: bash

$ mkdir pack
$ tar -zxf normaliz.tar.gz -C pack
$ cat > pack/entrypoint.sh <<- 'EOF'
#!/bin/sh
source bin/activate
conda-unpack
exec bin/normaliz $@
EOF
$ chmod +x pack/entrypoint.sh

Optional: reduce the size by removing files that are not needed here:

.. code-block:: bash

$ rm -rf pack/lib/*.a pack/usr/share pack/usr/include
$ find pack/lib -name '*.dylib' -type f -exec strip -S \{\} \; # macOS
$ find pack/lib -name '*.so' -type f -exec strip --strip-unneeded \{\} \; # Linux

Pack everything into a single shell script with `makeself <https://makeself.io/>`_:

.. code-block:: bash

$ conda install -y conda-forge::makeself
$ makeself pack/ normaliz.run Normaliz ./entrypoint.sh

The shell script `normaliz.run` now works on other machines with the same platform. Note that arguments to `bin/normaliz`
need to be given after an initial `--` since earlier arguments are consumed by makeself:

.. code-block:: bash

$ ./normaliz.run -- --version
Normaliz 3.8.5


Packaging a Complex Environment
-------------------------------

Complex environments can be packaged in the same way. Here we package
the computer algebra system `SageMath <https://sagemath.org>`_ which comes with a
Jupyter notebook interface:

.. code-block:: bash

$ conda create -y -n sagemath sage=9.2
$ conda pack -n sagemath
$ mkdir pack
$ tar -zxf sagemath.tar.gz -C pack
$ cat > pack/entrypoint.sh <<- 'EOF'
#!/bin/sh
source bin/activate
conda-unpack
exec bin/sage --notebook=jupyter $@
EOF
$ chmod +x pack/entrypoint.sh
$ makeself pack/ sagemath.run SageMath ./entrypoint.sh
$ ./sagmath.run # opens a browser with Jupyter running SageMath

The above creates a huge bundle that takes a long time to pack and unpack (and
might exceed the available space in your `/tmp`). This can be speeded up by
reducing the level of compression and by uncompressing things permanently:

.. code-block:: bash

$ cat > pack/unpack.sh <<- 'EOF'
#!/bin/sh
source bin/activate
conda-unpack
EOF
$ chmod +x pack/unpack.sh
$ cat > pack/sagemath.run <<- 'EOF'
#!/bin/sh
dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
cd "$dir"
./bin/sage --notebook=jupyter $@
EOF
$ chmod +x pack/sagemath.run
$ mkdir tmp
$ TMPDIR=tmp/ makeself --complevel 6 --target ./sagemath-9.2 pack/ sagemath.install SageMath ./unpack.sh

The resulting shell script unpacks the environment into `./sagemath-9.2`.
This can be overwritten with the `--target` parameter:

.. code-block:: bash

$ ./sagemath.install
$ ./sagemath-9.2/sagemath.run # opens a browser with Jupyter running SageMath