nixos/immich: init at 1.79.1

[oddlama]

Description of changes

This is an early draft to add a package and module for Immich, a very popular self-hosted photo and video backup solution. I would appreciate any feedback on module ergonomics. So far, all important settings are already exposed in the module and the packages should build correctly.

• Depends on nixos/typesense: init at 0.24.1 #244233
• Depends on python310Packages.insightface: init at 0.7.3 #244655 (currently includes a workaround that can be removed once this is merged)

TODOs:

• The nixos test would require network access and could benefit from persisting the machine learning models if that is possible (otherwise this is a multi-GB download each time the test is run)
• Uploading images / videos could invoke external tools such as ffmpeg, exiftool, ..., and at this point I haven't tested if all required tools are propagated.
• An update script is required to also update the sub-packages and their npmDepHashes.
• aarch64 support should be possible and should be tested.

Things done

Early draft, not everything can be tested yet.

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[AndersonTorres]

Split the commit

• one for the package
• one for nixos module
• one for module tests

[oddlama]

updated to 1.70.0, added automatic update script, split into separate commits

[qbit]

It seems nodePackages.immich is a thing, not sure if the -cli bit should be changed to use that or not?

[oddlama]

It seems nodePackages.immich is a thing, not sure if the -cli bit should be changed to use that or not?

That's the same thing that can be built from source here, just downloaded from the package registry. There currently are several cli implementations, and the one packaged here is the endorsed CLI that will replace all the others in the future.

[mattmelling]

immich-server looks for TYPESENSE_API_KEY even if TYPESENSE_ENABLE=false, so when cfg.server.typesense.enable = false, immich-server won't start.

[mattmelling]

I can get as far as the admin user sign up screen, hitting sign up causes an xhr to fail with a 500

[Atemu]

This needs a rebase as it does not build with the newer versions of its dependencies we have in master right now.

[oddlama]

Rebased and updated to 1.74.0. Beware that it doesn't build at this moment, because some transitive python dependencies are pulling in different versions of protobuf (which to my knowledge would require updates in nixpkgs before this will work again).

[Scrumplex]

I'm slowly starting to question whether we should bother packaging at the speed at which upstream moves...

I am personally very interested in this, as I don't really like my current Docker-Compose and previously Helm deployments of Immich.

At least upstream changes mostly reduce complexity, although the Postgres extension does feel a bit more annoying to implement than having a separate daemon

[oddlama]

Sorry for the slow update on this, I'm unfortunately quite busy at this time of year :/

I'm slowly starting to question whether we should bother packaging at the speed at which upstream moves...

I agree, this is becoming a concern more than I initially assumed. I've been in close contact with the immich devs for some time, so I've been aware of these big changes already, but they are still pushing out new stuff so insanely fast that it is hard to keep up.

Their architecture is just very much in motion right now and their development process (while very effective) is just docker-centric. That means there are quite some things in immich which assume exactly those conditions and that makes the module development a bit rough sometimes 😅. The current lack of E2E tests (which they are working on) means that I'll only discover these cases at runtime.

What concerns me most regarding this module is that each time they change something about their architecture (which is often) it will likely cause downstream breaking changes for users of this module.

I am personally very interested in this, as I don't really like my current Docker-Compose and previously Helm deployments of Immich.

Me too (obviously :P). But as long as native deployments are not a core goal of immich this will mean a lot of work and breakages on the NixOS side - and if I understand correctly that won't be the case for a long time, if ever.

Maybe the way to go forward here is to convert this draft to its own immich-nixos project that we can update more frequently and easily by ad-hoc packaging what is needed. That way we don't have to wait until all dependencies are available in NixOS proper. If anyone has their own thoughts on this matter, please share them :)

I'll definitely try to get all the changes in here sometime, but it'll take some work.

[christoph-heiss]

Maybe the way to go forward here is to convert this draft to its own immich-nixos project that we can update more frequently and easily by ad-hoc packaging what is needed. That way we don't have to wait until all dependencies are available in NixOS proper. If anyone has their own thoughts on this matter, please share them :)

That definitely sounds like a idea to pursue. I'd also would very much like to deploy/use Immich and be ready to help with such an effort.

You already listed a bunch of good reason for this, and having the possibility to move fast (just like upstream) would make things a lot easier IMHO. In addition to that, it'd enable much easier collaboration on the effort to keep up with upstream and provide a working version.

If, in the future, the rapid development with breakages all the time slows down a bit, it can still be integrated in nixpkgs after all.

Just my 2ct.

[diogotcorreia]

For those running Immich inside Docker but with PostgreSQL under NixOS, to update to 1.91.0 you need the pgvecto.rs PostgreSQL extension.

I've managed to (kinda) package it with the release binary (I did not try to build from source, seems to be a bit complicated because you need cargo-pgrx).
Here is the commit in my config updating to 1.91.0 if you want to copy the package: diogotcorreia/dotfiles@7676201
Suggestions are welcome!

[jvanbruegge]

pgvecto.rs is now in nixpkgs: #281192
Would be nice to have immich as well

[bhankas]

Since there seems to be an agreement on separate project being better suited for now, should we ask for a repo under nix-community? Ideally that would be better, but if we don't want that to be a blocker, we can always get started with any of the contributor's github and then transfer ownership. What say?

[Patryk27]

should we ask for a repo under nix-community

I think having flake.nix inside the official repository would be more useful, if upstream is willing.

[bhankas]

Since upstream devs don't use Nix for development (otherwise they'd already have a *.nix file), it would be undue burden on them to either keep the flake updated, or keep reviewing+accepting PRs for it, or provide someone full commit rights. I believe that is not a reasonable ask.

Besides, the idea behind separate repo for nixospackage+module is to be detached from both up streams (immich and nixpkgs) so we can keep working on the module itself.

Combined with f-droid allowing exact version of app to be and remain installed, both client and server can stay in sync until the package+module gets updated for upstream immich changes.

[adamcstephens]

Since there seems to be an agreement on separate project being better suited for now, should we ask for a repo under nix-community? Ideally that would be better, but if we don't want that to be a blocker, we can always get started with any of the contributor's github and then transfer ownership. What say?

Just create a repo, start building and invite maintainers if you want. No need for it to start under nix-community.

[pinpox]

Since there seems to be an agreement on separate project being better suited for now, should we ask for a repo under nix-community? Ideally that would be better, but if we don't want that to be a blocker, we can always get started with any of the contributor's github and then transfer ownership. What say?

Just create a repo, start building and invite maintainers if you want. No need for it to start under nix-community.

Feel free to ping or invite me, I'd like to test this.

[PowerUser64]

An example of this being done in the past is when the cemu emulator was made open-source https://github.com/yboettcher/CemuFlake.

The upstream project was changing fast because of growing pains (running on linux for the first time), and there was a naming conflict with cemu-ti (formerly just called cemu) in our repo, so this flake was made in the meantime. Sounds like similar action would work well here too.

[GetPsyched]

Have there been any updates on this effort?
cc @oddlama

[oddlama]

No sorry, currently I don't have this updated. Immich has been moving extremely fast recently, and they often introduce new dependencies which (especially with python ML stuff) requires a lot of work to package. I haven't had the time to keep it up with their pace.

If anyone wants to pick this up, feel free to do so. But be aware that this won't be a walk in the park to keep updated :P
Maybe I'll revisit this when immich goes stable.

[anoadragon453]

Here is my NixOS Immich setup in case it is useful to anyone here. I am currently running Immich v1.106.4.

https://github.com/anoadragon453/dotfiles/blob/main/modules/server/immich.nix

Note that to get pgvecto-rs to build (a postgres extension that Immich relies on) using current nixos-unstable, #320302 is needed. This PR is now merged and built on nixos-unstable.

[RomeoV]

Thanks @anoadragon453, this looks like it works for me. For others struggling to get it to work:

1. Copy Andrew's dotfile into e.g. /etc/nixos/immich.nix. If you typically use a stable release, change pkgs to pkgs_unstable and add a line

{config, lib, pkgs_unstable, ...}:
let
  pkgs = pkgs_unstable;
in 
...

3. Add something like this to your configuration.nix

{ config, lib, pkgs, ... }: 
let
  unstable = import <nixos-unstable> {};
  pkgs_unstable = unstable.pkgs;
in {
  imports =   [
    (import /etc/nixos/immich.nix {inherit lib config pkgs_unstable; })
  ];

  services.immich = {
    enable = true;
    domain = "immich.your-domain.xyz";
    port = 8006;
    metricsPortServer = 8009;
    metricsPortMicroservices = 8010;
    storagePath = "/mnt/storage-box/immich";
    logLevel = "log";
  };

4. et voilà

[jvanbruegge]

If someone does not want to run docker containers for immich and would like a native nixos package instead, I've packaged the current version (v1.106.4) and also built a nixos module.

If you use flakes, you should be able to include the module like this:

{
  description = "NixOS configuration";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    immich.url = "github:jvanbruegge/server-config";
    immich.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = inputs@{ nixpkgs, immich, ... }: {
    nixosConfigurations = {
      hostname = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          ./configuration.nix
          immich.nixosModules.immich
        ];
      };
    };
  };
};

You can then use it like this:

services.immich = {
  enable = true;
  host = "0.0.0.0";
  port = 8080;
  mediaLocation = "/mnt/storage-box/immich";
  openFirewall = true;
  secretsFile = "/run/secrets/immich"; # This has at least DB_PASSWORD=<pass>
};
sops.secrets.immich = {};

The flake also exports just the package (without the module). I have this running for a few weeks now on my own server.

[Atemu]

@jvanbruegge Would you be willing to create a PR for the package and optionally the module too?

[marcusramberg]

The flake also exports just the package (without the module). I have this running for a few weeks now on my own server.

https://github.com/jvanbruegge/server-config/blob/5ee96717d8476c6671bb1c28d144007bdf74915b/flake.nix#L41 seems like your flake currently does not export the module at all?

[jvanbruegge]

seems like your flake currently does not export the module at all?

Sorry about that, that got lost in rebasing. Fixed now.

Would you be willing to create a PR for the package and optionally the module too?

Sure, I can do so soon

[Pablito2020]

seems like your flake currently does not export the module at all?

Sorry about that, that got lost in rebasing. Fixed now.

Would you be willing to create a PR for the package and optionally the module too?

Sure, I can do so soon

You need to remove the "self" attribute, just like I did in this commit.

I also noticed that you use your own DB instead of the one declared in your module. This works on your machine, but the default password in the module DB is empty, which Immich doesn't allow. I added a default password in this commit. This is a quick fix; we can later get it from the module input.

You can test it with the following configuration:

  services.immich = {
    enable = true;
    host = "0.0.0.0";
    port = 8080;
    mediaLocation = "/data/immich";
    openFirewall = true;
    environment = {
      DB_USERNAME="immich";
      DB_PASSWORD="password123";
    };
    database.createDB = true;
    secretsFile = config.age.secrets.immichDatabase.path; # Right now this file is empty, since we pass the DB_PASSWORD in environment... Dirty hack as I said.
  };

Additionally, we need to fix the permissions of the mediaLocation folder. I had to run:

$ chmod -R 0777 /data

And voilà! Immich is working on my machine.

Thanks a lot for the module! Coming from your dotfiles, I only needed a bit of tweaking to get it working. Making it an official module won't take much time, though maintaining it might be challenging since Immich evolves quickly and with breaking changes.

Cheers.

[christoph-heiss]

@Pablito2020

$ chmod -R 0777 /data

Please never do that! Are you aware of the consequences?
It should be owned by the immich user & group only, without allowing every users accessing it.

[Pablito2020]

@Pablito2020

$ chmod -R 0777 /data

Please never do that! Are you aware of the consequences? It should be owned by the immich user & group only, without allowing every users accessing it.

Yes, I am aware of the consequences of giving rwx access to all users of the machine, it was just for getting it to work, someone should add the privileges to the systemd-service inside the module.

I was just trying to tell what's missing inside the module for making it work on the first try.

I am sorry if my comment wasn't clear.

[christoph-heiss]

Derailed into OT now completely, but:

Yes, I am aware of the consequences of giving rwx access to all users of the machine, it was just for getting it to work, someone should add the privileges to the systemd-service inside the module.

It is just that people really like to blindly run random commands people post on the internet without knowing what they do, inevitable resulting in losing data or such in the end ..

Sometimes you (unfortunately!) have to protect people from themselves.

[jvanbruegge]

I've now opened the PR for the package and module here: #324127

Differences to the version in my repo:

• Updated the version to v1.107.1 released a few minutes ago
• The cli executable is just called immich as that is what the immich docs use
• All configuration options in the module have documentation now
• The postgres config by default uses the unix socket now, meaning that services.immich.enable = true is enough to get a working service.
• I've added a nixos test that creates an admin user, creates an API key for that admin user and then uses the immich-cli to get the server-info (so that both server and cli get tested)