terraform: add var.extra_build_env_vars, closes #413

[KiaraGrouwstra]

This PR adds a Terraform input variable I so far named extra_build_env_vars (suggestions welcome).
This allows passing in a map of attributes from Terraform to expose to nixos-anywhere's build step to make them available as environment variables within a NixOS configuration.

For this implementation I opted to leave actually passing the info to the user, so people stay in control over what info gets exposed where.
I opted to focus on allowing to pass info thru environment variables here figuring that would be less prone to fighting flakes over what is staged to Git, as committing info to be passed here may be undesirable both for ephemeral noise (if not also over sensitive info - tho any info passed this way lands in /nix/store/ as well).

To pull this off I used --impure plus eval - feedback on this would be great.

On the Terraform side as a consumer, to best enjoy this I have had to resort to some more involved logic as well, that gets easy when using Terraform from Nix. As an example (based on Hetzner Cloud), using the lib.tfRef function from Terranix I would use something like:

let
  servers = ...;
  variable = ...;
  data = ...;
  resource = ...;
in
{
  inherit variable data resource;
  module =
    lib.mapAttrs (server_name: _server_config: let
    in {
      # pin module version by nix flake inputs
      source = "github.com/numtide/nixos-anywhere?ref=${inputs.nixos-anywhere.sourceInfo.rev}/terraform/all-in-one";
      ...
      extra_build_env_vars = {
        # all variables
        # TF_VARS = lib.strings.toJSON (lib.mapAttrs (k: _: lib.tfRef "jsonencode(var.${k})") variable);
        # non-sensitive variables
        TF_VARS = lib.tfRef "jsonencode(${lib.strings.toJSON (lib.mapAttrs (k: _: lib.tfRef "var.${k}") (lib.filterAttrs (_k: v: !(v ? sensitive && v.sensitive)) variable))})";
        TF_DATA = lib.tfRef "jsonencode(${lib.strings.toJSON (lib.mapAttrs (type: instances: lib.mapAttrs (k: _: lib.tfRef "data.${type}.${k}") instances) data)})";
        TF_RESOURCES = lib.tfRef "jsonencode(${lib.strings.toJSON (lib.mapAttrs (type: instances: lib.mapAttrs (k: _: lib.tfRef "resource.${type}.${k}") instances) resource)})";
        TF_SERVER = lib.tfRef "jsonencode(resource.hcloud_server.${server_name})";
        SERVER_NAME = server_name;
      };
    })
    servers;
}

You can then verify their contents from your nixosConfigurations like:

          lib.nixosSystem {
            inherit system;
            modules = [
              {
                environment.etc = {
                  TF_VARS.text = builtins.getEnv "TF_VARS";
                  TF_DATA.text = builtins.getEnv "TF_DATA";
                  TF_RESOURCES.text = builtins.getEnv "TF_RESOURCES";
                  TF_SERVER.text = builtins.getEnv "TF_SERVER";
                  SERVER_NAME.text = builtins.getEnv "SERVER_NAME";
                };
              }
            ];
          };

-> ls /etc/ -> cat /etc/TF_SERVER

(in practice one would instead probably parse these back to nix values to grab specific properties from, e.g. like: lib.strings.fromJSON (builtins.getEnv "TF_SERVER"))

[Mic92]

Mhm --impure by default is not so nice.

Have you considered the following approach?

https://github.com/NixOS/nixos-wiki-infra/blob/main/terraform/nixos-wiki/nixos_vars.tf

Basically one can use terraform to generate a json file and using git add to make sure it's recognized by the flake. Maybe we can encode this pattern into the terraform module instead of using builtins.getEnv?
The big upside on this is that, you are no longer forced to always have to use terraform to build or deploy your nixos configuration.

[Mic92]

Just imagine you want your machines to be build by CI. Before you could just do nixos-rebuild build --flake .mymachine, now you have to do some sort of terraform plan.

[Mic92]

Interface wise I would propose a nix-file module that would use git add --intent-to-add --force -- ./file >/dev/null 2>&1 || true to make sure that it is added to your git repository (if one exists). It would take content as a parameter:

module "nixos_vars" { 
   source  = "../nix-file"
   content = jsonencode({})
   filename = "${path.module}/nixos-vars.json"
}

[KiaraGrouwstra]

hm. if one could make a file without terraform, why couldn't one similarly make env vars, if the nixos configuration is indeed made to require them?

locally such a nix-file module might give some noise in version control, tho one could opt to just never use git add . anymore. whereas deploying from CI might circumvent that, it would seem a bit annoying in a development setting. going by file may have use-cases as well, tho it'd be nice to see a route more amenable to the common use-case of git + flakes + development supported as well.

on --impure, maybe a conditional could be added.

[Mic92]

As we discussed it's possible to achieve your goal without impure evaluation. Check how to import json into a nixos configuration: https://github.com/nix-community/disko/blob/09a776702b004fdf9c41a024e1299d575ee18a7d/install-cli.nix#L56

disko-install still adds --impure. However it's possible to use getFlake also in a pure evaluation: Check out this code in clan: https://git.clan.lol/clan/clan-core/src/commit/bcf2cd1814bbef57c74b0b4eec8f6a7cc0479475/pkgs/clan-cli/clan_cli/machines/machines.py#L229

[Mic92]

KiaraGrouwstra@75e09d1 also looks good.

[KiaraGrouwstra]

@Mic92 thanks for elaborating - i originally hadn't quite figured out the tempfile thing. i'll try and see if I can use your mentioned approach to resolve the flake+git friction.

[KiaraGrouwstra]

notes to self to reconcile the PRs, before i forget: info may be passed purely without staging to git thru nix build --expr, reading a file by builtins.fetchTree to pass the content as file to avoid stack overflows based on content size. the original lib.nixosSystem can then be wrapped so as to e.g. write the extra info to some file.

edit: potential alternative: nix build path:.

[Mic92]

I think in general it would be fine to have environment variables or flags passed to nixos-rebuild. This could be used for injecting binary cache configuration.

[KiaraGrouwstra]

thanks, i'll check it out. i still got a few question marks around say how that would handle conditionals based on environment variables, but i guess i can just try that.

[Mic92]

This is obsolete now.