Support declarative container options in imperative setup

[danbst]

It is now impossible to set bind mounts for imperative containers. I don't whether it is by design, or there is no interface to do it imperatively yet. I hope for latter. In that case we can have this container config:

{ config, pkgs, lib, ...}:
{
   services.postgresql.enable = true;
   services.postgresql.dataDir = "/media/db";
   deployment.container.bindMounts = [{ "/media/db" = { hostPath = "/media/db"; isReadOnly = false; }; }];
   deployment.container.privateNetwork = true;
}

Main use case is to improve usability of container deployments in NixOps. Another usecase it to simplify solution for problem in #17365 (if I understood the problem correctly).

[groxxda]

Imperative bind mounts should be possible via machinectl bind <machine> <hostpath> [<containerpath>] optionally with --mkdir to create the path inside the container.

I doubt they persist across reboots but have not tested.

If you are looking for a declarative way, this works:

containers.<machine>.bindMounts."<containerpath>".hostPath = "<hostpath>";

[johbo]

Came around this problem while trying to deploy containers via NixOPS. I've done some early experiments.

Declarative containers are currently quite powerful compared to the imperative approach and that's a pity, since NixOPS' container backend is based on the imperative approach via the tool nixos-container.

From researching how containers are actually implemented, I've found that declarative containers take advantage of custom unit files for some settings and they use mode values in the container's configuration file. These values are then used in the startup files of the container.

Imperative containers have way less options for configuration. They use a template unit file based on an empty dummy configuration and the container's configuration file is written by nixos-container instead of being built via Nix.

By manually tweaking the configuration file of my test containers in /etc/containers/ some of the features from declarative containers can be used. They are just not accessible via nixos-container, still they work. That's because the startup scripts of the containers base most of their decisions on this configuration file.

Most of the fragments are defined in this module: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/containers.nix

Based on my current understanding I think that there might be a chance to actually adjust the way how containers are configured, so that the deployment options sit inside of the container definition, like you suggest above. In my experiments I've used a different namespace config.container instead of deployment.container just because the deployment namespace seems to be more related to nixops. I might be wrong here though.

The idea is to move all options of the declarative containers from containers.<name>.* into containers.<name>.config.* so that they would reside inside the container's configuration. This way we could also use them in NixOPS and probably also for imperative containers based on a configuration file via nixos-container.

Currently I am thinking that the container's configuration file (or in the imperative case the base of it) should be build by nix and maybe added as a symlink into the system derivation, so that nixos-container could use it from there and the user could inspect it via the system profile. I am not sure if that's having unwanted side effects though, first time I am touching this area.

It would probably also be possible to support adjusted startup scripts based on the same logic which is used for declarative containers. The tricky part would be to tell the template unit file, that we want a specific startup script or snippet to be executed. This might be doable based on a new variable in the container's configuration file, so that the unit template of the host would find the script.

I've made an experiment which demonstrates parts of the idea. Would be great to get some feedback if that's a reasonable direction. For simplicity I've put the code into my nixops fork into this branch: https://github.com/johbo/nixops/tree/exp-containers

The compare view can be found here: https://github.com/NixOS/nixops/compare/master...johbo:exp-containers?expand=1

The following is an example how I've used it in my small test scenario:

  cont1 = { config, pkgs, ... }: {
    # NixOPS specific
    deployment.targetEnv = "container";
    deployment.container.host = "nixvm1";

    # This would also work in NixOS, needs a tweaked nixos-container script though.
    container = {
      writeContainerConfig = true;
      privateNetwork = true;
      localAddress = "10.233.0.2";
      hostAddress = "10.233.0.1";
      hostBridge = "br0";
    };
  };

[danbst]

Looking forward for Nixos side implementation! Why did you go nixops side?

[boxofrox]

I stumbled upon this while trying to learn how to define bind mounts in an imperative container. I tried looking at @johbo's commits, but I'm new to Nix and it's all gibberish to me at this point.

I want to mention a concern that @johbo's commits may or may not take into account (if it even should). I noticed that I'm editing settings for my imperative container in /var/lib/containers/<name>/etc/nixos/configuration.nix, and this file appears to be inside the container. If bindMounts and such are specified in this config file, is there no concern that a compromised container might edit the configuration.nix, change the bindMounts, and trigger a restart of the container by killing all processes, and gain access to portions of the filesystem that should never have been accessible?

Declarative containers, on the other hand, do not have a /var/lib/containers/<name>/etc/nixos/configuration.nix, AFAICT, and if the settings aren't exposed inside the container in a writable fashion, I find this preferable.

Perhaps dropping privileges inside the container is sufficient (I still need to determine how this is achieved), or perhaps Nix is so awesome it already handles this, but if configuring bindMounts inside a container is the direction this solution is heading, consider moving the configuration.nix outside the container (or lock down writes even from root) as you move forward.

[danbst]

Currently it is clearly stated that compromised root in container can get access to root in host, but your concern is a valid one

[nh2]

I need this too, because I need to override DeviceAllow for my nixops containers. See NixOS/nixops#802

[danbst]

My usecase is finally possible with https://github.com/erikarvstedt/extra-container

Next issue should be intergrate extra-container into nixos-container.

[nh2]

@danbst Did you inted to close this? It seems like until

intergrate extra-container into nixos-container

is done, this problem continues to exist.

(Tentatively reopening)

[danbst]

yep, I wanted to close this issue. Actually, I think integrating extra-container stuff into nixos-container isn't trivial, maybe better would be to design nixos container command from scratch (#54188), however that isn't trivial as well =)

[nh2]

integrating extra-container stuff into nixos-container isn't trivial, maybe better would be to design nixos container command from scratch (#54188), however that isn't trivial as well =)

@danbst That all sounds reasonable, but shouldn't we keep the ticket open then, as there is no current general solution?

[danbst]

@nh2 alright. I wonder if there are drawbacks of using deployment.container.* option inside container config versus containers.* outside container config (apart from implementation complications).

[ghost]

Nix is now, with flakes, going towards splitting code up. Merging everything in nixpkgs isn't the way to go anymore. Containers are, in all likelyhood, better as a separate flake. We can go the other way, and take stuff out of nixpkgs.

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[nh2]

still important to me

[stale]

I marked this as stale due to inactivity. → More info

[aanderse]

I wonder what @erikarvstedt has to say on this topic...

[stale]

I marked this as stale due to inactivity. → More info

[andrevmatos]

also interested in this, specially with flakes

[stale]

I marked this as stale due to inactivity. → More info

[ranfdev]

Still interested, for containers defined inside flakes

[ranfdev]

It's worth taking a look at microvm, which is setting up vm options as nixos modules:

outputs = { self, nixpkgs, microvm }: {
    # Example nixosConfigurations entry
    nixosConfigurations.my-microvm = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        # Include the microvm module
        microvm.nixosModules.microvm
        # Add more modules here
        {
          networking.hostName = "my-microvm";
          microvm.hypervisor = "cloud-hypervisor";
        }
      ];
    };
  };

It should be possible to create a module defining container options, to be put inside the container definition, as microvm is doing