Nix helpers for Clojure projects
STATUS: alpha. Please leave feedback.
The main goal of the project is to reduce the friction between Clojure and Nix. Nix is a great tool to build and deploy software, but Clojure is not well supported in the Nix ecosystem.
clj-nix
tries to improve the situation, providing Nix helpers to interact with
Clojure projects
The main difficulty of packaging a Clojure application with Nix is that the derivation is restricted from performing any network request. But Clojure does network requests to resolve the dependency tree. Some network requests are done by Maven, since Clojure uses Maven under the hood. On the other hand, since git deps were introduced, Clojure also access the network to resolve the git dependencies.
A common solution to this problem are lock files. A lock file is a snapshot of the entire dependency tree, usually generated the first time we install the dependencies. Subsequent installations will use the lock file to install exactly the same version of every dependency. Knowing beforehand all the dependencies, we can download and cache all of them, avoiding network requests during the build phase with Nix.
Ideally, we could reuse a lock file generated by Maven or Clojure itself, but
lock files are not popular in the JVM/Maven ecosystem. For that reason,
clj-nix
provides a way to create a lock file from a deps.edn
file. Creating
a lock file is a prerequisite to use the Nix helpers provided by clj-nix
GOALS:
- Create a binary from a clojure application
- Create an optimized JDK runtime to execute the clojure binary
- Create GraalVM native images from a clojure application
- Simplify container creation for a Clojure application
- Run any arbitrary clojure command at Nix build time (like
clj -T:build
orclj -M:test
)
This project requires Nix Flakes
nix flake new --template github:jlesquembre/clj-nix ./my-new-project
cd ./my-new-project
git init
git add .
Remember that with flakes, only the files tracked by git are recognized by Nix.
Templates are for new projects. If you want to add clj-nix
to an existing
project, I suggest just copy the parts you need from the template (located here:
clj-nix/templates/default)
As mentioned, a lock file must be generated in advance:
nix run github:jlesquembre/clj-nix#deps-lock
That command looks for deps.edn
files in your project and generates a
deps-lock.json
file in the current directory. Remember to re-run it if you
update your dependencies.
It is possible to add the dependencies to the nix store during the lock file
generation. Internally we are invoking the nix store add-path
command. By
default, it's disabled because that command is relatively slow. To add the
dependencies to the nix store, set the environment variable
CLJNIX_ADD_NIX_STORE
to true, e.g.:
CLJNIX_ADD_NIX_STORE=true nix run github:jlesquembre/clj-nix#deps-lock
Sometimes it could be useful to ignore some deps.edn
files, to do that, just
pass the list of files to ignore the the deps-lock
command:
nix run github:jlesquembre/clj-nix#deps-lock -- ignore/deps.edn
Leiningen projects are supported. Use the --lein
option to add the
project.clj
dependencies to the lock file. This option can be combined with
ignored files:
nix run github:jlesquembre/clj-nix#deps-lock -- --lein ignore/deps.edn
Keep in mind that deps-lock
command is not optimized for Leiningen projects,
it will download all the maven dependencies every time we generate the lock
file. For that reason, it is recommended to add a deps.edn
file with the same
dependencies to Leiningen projects. That way, we reduce the number of network
requests when the deps-lock
command is invoked.
There are projects to automatically generate a deps.edn
file from a Leiningen
project (e.g.: depify)
Derivations:
- mkCljBin: Creates a clojure application
- customJdk: Creates a custom JDK with jlink. Optionally takes a
derivation created with
mkCljBin
. The intended use case is to create a minimal JDK you can deploy in a container (e.g: a Docker image) - mkGraalBin: Creates a binary with GraalVM from a derivation
created with
mkCljBin
- mkCljLib: Creates a clojure library jar
- mkBabashka: Builds custom babashka
NOTE: Extra unknown attributes are passed to the mkDerivation
function,
see mkCljBin section for an example about how to add a custom check
phase.
Helpers:
- mkCljCli: Takes a derivation created with
customJdk
and returns a valid command to launch the application, as a string. Useful when creating a container. - bbTasksFromFile: Helper to wrap all the clojure functions in a file as bash scripts. Useful to create a nix develpment shell with devshell
- mk-deps-cache: Creates a Clojure deps cache (maven cache +
gitlibs cache). Used by
mkCljBin
andmkCljLib
. You can use this function to to have access to the cache in a nix derivation.
Creates a Clojure application. Takes the following attributes (those without a default are mandatory, extra attributes are passed to mkDerivation):
-
jdkRunner: JDK used at runtime by the application. (Default:
jdk
) -
projectSrc: Project source code.
-
name: Derivation and clojure project name. It's recommended to use a namespaced name. If not, a namespace is added automatically. E.g.
foo
will be transformed tofoo/foo
-
version: Derivation and clojure project version. (Default:
DEV
) -
main-ns: Main clojure namespace. A
-main
function is expected here. -
buildCommand: Command to build the jar application. If not provided, a default builder is used: build.clj. If you provide your own build command, clj-nix expects that a jar will be generated in a directory called
target
-
lockfile: The lock file. (Default:
${projectSrc}/deps-lock.json
)
Example:
mkCljBin {
jdkRunner = pkgs.jdk17_headless;
projectSrc = ./.;
name = "me.lafuente/clj-tuto";
version = "1.0";
main-ns = "demo.core";
buildCommand = "clj -T:build uber";
# mkDerivation attributes
doCheck = true;
checkPhase = "clj -M:test";
}
Outputs:
- out: The application binary
- lib: The application jar
Creates a custom JDK runtime. Takes the following attributes (those without a default are mandatory):
-
jdkBase: JDK used to build the custom JDK with jlink. (Default:
nixpkgs.jdk17_headless
) -
cljDrv: Derivation generated with
mkCljBin
. -
name: Derivation name. (Default:
cljDrv.name
) -
version: Derivation version. (Default:
cljDrv.version
) -
jdkModules: Option passed to jlink
--add-modules
. If null,jeps
will be used to analyze thecljDrv
and pick the necessary modules automatically. (Default:null
) -
multiRelease: Option passed to jdeps
--multi-release
. Should be an integer >=9 or a boolean. If true, the value is set tobase
. If false or not specified, clj-nix will try to detect if the jar is a multi-release jar and set the value automatically. See The jdeps Command for more info. (Default:false
) -
locales: Option passed to jlink
--include-locales
. (Default:null
)
Example:
customJdk {
jdkBase = pkgs.jdk17_headless;
name = "myApp";
version = "1.0.0";
cljDrv = myCljBinDerivation;
locales = "en,es";
}
Outputs:
- out: The application binary, using the custom JDK
- jdk: The custom JDK
Generates a binary with GraalVM from an application created with mkCljBin
.
Takes the following attributes (those without a default are mandatory):
-
cljDrv: Derivation generated with
mkCljBin
. -
graalvm: GraalVM used at build time. (Default:
nixpkgs.graalvmCEPackages.graalvm17-ce
) -
name: Derivation name. (Default:
cljDrv.name
) -
version: Derivation version. (Default:
cljDrv.version
) -
extraNativeImageBuildArgs: Extra arguments to be passed to the native-image command. (Default:
[ ]
) -
graalvmXmx: XMX size of GraalVM during build (Default:
"-J-Xmx6g"
)
Example:
mkGraalBin {
cljDrv = myCljBinDerivation;
}
An extra attribute is present in the derivation, agentlib
, which generates a
script to help with the generation of a reflection config file
Creates a jar file for a Clojure library. Takes the following attributes (those without a default are mandatory, extra attributes are passed to mkDerivation):
-
projectSrc: Project source code.
-
name: Derivation and clojure library name. It's recommended to use a namespaced name. If not, a namespace is added automatically. E.g.
foo
will be transformed tofoo/foo
-
version: Derivation and clojure project version. (Default:
DEV
) -
buildCommand: Command to build the jar application. If not provided, a default builder is used: jar fn in build.clj. If you provide your own build command, clj-nix expects that a jar will be generated in a directory called
target
Example:
mkCljLib {
projectSrc = ./.;
name = "me.lafuente/my-lib";
buildCommand = "clj -T:build jar";
}
Builds Babashka with the specified
features. See
babashka feature flags
for the full list. Notice that the feature names in the Nix wrapper are case
insensitive and we can omit the BABASHKA_FEATURE_
prefix.
Takes the following attributes:
-
withFeatures: List of extra Babashka features. (Default:
[]
) -
bbLean: Disable default Babashka features. (Default:
false
) -
graalvm: GraalVM used at build time. (Default:
nixpkgs.graalvmCEPackages.graalvm11-ce
)
Example:
mkBabashka {
withFeatures = [ "jdbc" "sqlite" ];
}
Returns a string with the command to launch an application created with
customJdk
. Takes the following attributes (those without a default are
mandatory):
jdkDrv: Derivation generated with customJdk
java-opts: Extra arguments for the Java command (Default: []
)
extra-args: Extra arguments for the Clojure application (Default: ""
)
Example:
mkCljCli {
jdkDrv = self.packages."${system}".jdk-tuto;
java-opts = [ "-Dclojure.compiler.direct-linking=true" ];
extra-args = [ "--foo bar" ];
}
Reads a Clojure file, for each function generates a devshell command. Takes a path (to a Clojure file) or the following attributes:
file: Path to the Clojure file
bb: Babashka derivation (Default: nixpkgs.babashka
)
Example:
devShells.default =
pkgs.devshell.mkShell {
commands = pkgs.bbTasksFromFile ./tasks.clj;
# or
commands = pkgs.bbTasksFromFile {
file = ./tasks.clj;
bb = pkgs.mkBabashka { withFeatures = [ "jdbc" "sqlite" ]; };
};
}
Generate maven + gitlib cache from a lock file. This is a lower level helper,
usually you want to use mkCljBin
or mkCljLib
and define a custom build
command with the buildCommand
argument.
lockfile: deps-lock.json file
Example:
mk-deps-cache {
lockfile = ./deps-lock.json;
}
It's possible to add a GitHub action to automatically update the
deps-lock.json
file on changes:
name: "Update deps-lock.json"
on:
push:
paths:
- "**/deps.edn"
jobs:
update-lock:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: cachix/install-nix-action@v17
- name: Update deps-lock
run: "nix run github:jlesquembre/clj-nix#deps-lock"
- name: Create Pull Request
uses: peter-evans/[email protected]
with:
commit-message: Update deps-lock.json
title: Update deps-lock.json
branch: update-deps-lock
- How can I define extra runtime dependencies?
If you try to call a external program from your Clojure application, you will
get an error (something like
Cannot run program "some-program": error=2, No such file or directory
). One
possible solution is to wrap the binary:
cljpkgs.mkCljBin {
# ...
nativeBuildInputs = [ pkgs.makeWrapper ];
postInstall = ''
wrapProgram $cljBinary \
--set PATH ${pkgs.lib.makeBinPath [ pkgs.cowsay ]}
'';
}
Notice that the $cljBinary
is a proper Bash variable. It is created by
mkCljBin
during the install phase.
or if you want to define the dependencies in a docker image:
pkgs.dockerTools.buildLayeredImage {
# ...
config = {
Env = [ "PATH=${pkgs.lib.makeBinPath [ pkgs.cowsay ]}" ];
};
};
Source code for this tutorial can be found here: https://github.com/jlesquembre/clj-demo-project
There is a template to help you start your new project:
nix flake new --template github:jlesquembre/clj-nix ./my-new-project
For this tutorial you can clone the final version:
git clone [email protected]:jlesquembre/clj-demo-project.git
First thing we need to do is to generate a lock file:
nix run github:jlesquembre/clj-nix#deps-lock
git add deps-lock.json
NOTE: The following examples assume that you cloned the demo repository, and
you are executing the commands from the root of the repository. But with Nix
flakes, it's possible to point to the remote git repository. E.g.: We can
replace nix run .#foo
with nix run github:/jlesquembre/clj-demo-project#foo
First, we create a new package in our flake:
clj-tuto = cljpkgs.mkCljBin {
projectSrc = ./.;
name = "me.lafuente/cljdemo";
main-ns = "demo.core";
};
Let's try it:
nix build .#clj-tuto
./result/bin/clj-tuto
# Or
nix run .#clj-tuto
Nice! We have a binary for our application. But how big is our app? We can find it with:
nix path-info -sSh .#clj-tuto
# Or to see all the dependencies:
nix path-info -rsSh .#clj-tuto
Um, the size of our application is 1.3G
, not ideal if we want to create a
container. We can use a headless JDK to reduce the size, let's try that:
clj-tuto = cljpkgs.mkCljBin {
projectSrc = ./.;
name = "me.lafuente/cljdemo";
main-ns = "demo.core";
jdkRunner = pkgs.jdk17_headless;
};
nix build .#clj-tuto
nix path-info -sSh .#clj-tuto
Good, now the size is 703.9M
. It's an improvement, but still big. To reduce
the size, we can use the customJdk
helper.
We add a package to our flake, to build a customized JDK for our Clojure application:
jdk-tuto = cljpkgs.customJdk {
cljDrv = self.packages."${system}".clj-tuto;
locales = "en,es";
};
nix build .#jdk-tuto
nix path-info -sSh .#jdk-tuto
Not bad! We reduced the size to 96.3M
. That's something we can put in a
container. Let's create a container with our application.
Again, we add a new package to our flake, in this case it will create a container:
clj-container =
pkgs.dockerTools.buildLayeredImage {
name = "clj-nix";
tag = "latest";
config = {
Cmd = clj-nix.lib.mkCljCli self.packages."${system}".jdk-tuto { };
};
};
nix build .#clj-container
nix path-info -sSh .#clj-container
The container's size is 52.8M
. Wait, how can be smaller than our custom JDK
derivation? There are 2 things to consider.
First, notice that we used the mkCljCli
helper function. In the original
version, our binary is a bash script, so bash
is a dependency. But in a
container we don't need bash
, the container runtime can launch the command,
and we can reduce the size by removing bash
Second, notice that the image was compressed with gzip.
Let's load and execute the image:
docker load < result
docker run -it --rm clj-nix
docker images
Docker reports an image size of 99.2MB
If we want to continue reducing the size of our derivation, we can compile the application with GraalVM. Keep in mind that size it's not the only factor to consider. There is a nice slide from the GraalVM team, illustrating what technology to use for which use case:
(The image was taken from a tweet by Thomas Würthinger)
For more details, see: Does GraalVM native image increase overall application performance or just reduce startup times?
Let's compile our Clojure application with GraalVM:
graal-tuto = cljpkgs.mkGraalBin {
cljDrv = self.packages."${system}".clj-tuto;
};
nix build .#graal-tuto
./result/bin/clj-tuto
nix path-info -sSh .#graal-tuto
The size is just 43.4M
.
We can create a container from this derivation too:
graal-container =
let
graalDrv = self.packages."${system}".graal-tuto;
in
pkgs.dockerTools.buildLayeredImage {
name = "clj-graal-nix";
tag = "latest";
config = {
Cmd = "${graalDrv}/bin/${graalDrv.name}";
};
};
docker load < result
docker run -it --rm clj-graal-nix
In this case, the container image size is 45.3MB
, aproximately half the size
of the custom JDK image.