Emacs calls doom incorrectly

[thecaralice]

Commands like SPC-h-r-r call the original doom with EMACS pointing to the original Emacs, which causes obscure issues. I patched doomemacs while debugging an issue I had to call my debugging utility instead of doom, here's the result:

Arguments: [
    "mole", # this is the argv[0]
    "/nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source/bin/doom", # this would be the argv[0]
    "sync",
    "-B",
    "-e",
]

Environment: {
    "DOOMDIR": "/Users/alice/.config/doom/",
    "DOOMLOCALDIR": "/Users/alice/.local/share/nix-doom/",
    "DOOMPROFILE": "nix",
    "DOOMPROFILELOADFILE": "/nix/store/r5xp99h6bs20cb5d3b49xk6jarnvj0d9-doom-profile/loader/init.el",
    "EMACS": "/nix/store/ysr370j2nylqbvwmnk2y3w2hfm0w09ky-emacs-pgtk-29.4/bin/emacs",
    "EMACSDIR": "/nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source/",
    "EMACSLOADPATH": "",
    "EMACSNATIVELOADPATH": "",
    "INSIDE_EMACS": "29.4,comint",
    "OLDPWD": "/nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source",
    "PATH": "/nix/store/wixmy825ga0mzv5ic5k7p471n8760c8b-mole-0.0.1/bin:/Users/alice/.nix-profile/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin",
    "PWD": "/nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source",
    # irrelevant variables omitted
}

Passing these EMACS and DOOMLOCALDIR to that doom executable is enough to reproduce the issue I had with SPC-h-r-r:

$ DOOMLOCALDIR=/Users/alice/.local/share/nix-doom/ EMACS=/nix/store/ysr370j2nylqbvwmnk2y3w2hfm0w09ky-emacs-pgtk-29.4/bin/emacs /nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source/bin/doom sync -B -e
- Using Emacs 29.4 @ /nix/store/ysr370j2nylqbvwmnk2y3w2hfm0w09ky-emacs-pgtk-29.4/bin/emacs
> Synchronizing "default" profile...
  x There was an unexpected runtime error
    Message: File is missing
    Details: ("Cannot open load file" "No such file or directory" "straight")
    Backtrace:
      (require straight)
      (nix-doom-skip-core-packages nil)
      (apply nix-doom-skip-core-packages nil)
      (doom-initialize-core-packages nil)
      (doom-initialize-packages)
      (doom-packages-ensure nil)
      (let ((doom-print-level (or nil doom-print-level)) (doom-print-indent ...
      (progn (let ((doom-print-level (or nil doom-print-level)) (doom-print-...
      (unwind-protect (progn (let ((doom-print-level (or nil doom-print-leve...
      (let ((noenvvar? (cdr (assq 'noenvvar? alist))) (update? (cdr (assq 'u...
      ((closure (t) (cli alist) (let ((noenvvar? (cdr (assq 'noenvvar? alist...
      (funcall (closure (t) (cli alist) (let ((noenvvar? (cdr (assq 'noenvva...
GNU Emacs     v29.4            nil
Doom core     v3.0.0-p
To add an exception for this directory, call:

        git config --global --add safe.directory /nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source
Doom modules  v24.11.0-pre     fatal: detected dubious ownership in repository at '/nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source'
To add an exception for this directory, call:

        git config --global --add safe.directory /nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source
    ! Wrote extended backtrace to ~/.local/share/nix-doom/state/logs/cli.doom.241103232114.98372.error
✓ Finished in 0.63710s

Note: /nix/store/09391jc8xazy2mrjhmca0xzblrmxm9iz-source/ is the doomSource and /nix/store/ysr370j2nylqbvwmnk2y3w2hfm0w09ky-emacs-pgtk-29.4/ is the original Emacs I use for programs.doom-emacs.emacs (pkgs.emacs29-pgtk)

Possible solution

Change --init-directory to a new directory (probably just a symlinkJoin) which would have the wrapped doom.

[marienz]

Couple of things going on here...

Question: what do you want/expect doom/reload (SPC h r r) to do?

In vanilla Doom, you'd usually use this to test changes you made in your (mutable) $DOOMDIR / doom-user-dir. But in our case, all relevant paths point into the store: the best you're likely to get is "reload your original configuration" (the one you built the running doom-emacs with).

Is that what you want / expect? I don't want to spend too much time trying to make this not error out if the end result isn't going to be useful, and if that is what you want I'd be inclined to override doom/reload with something that just reloads config.el files (both for Doom modules and your private config).

Background:

First, doom sync. In normal Doom, that would read your init.el and packages.el from your $DOOMDIR, install/update packages accordingly, and regenerate the profile init file.

This makes sense if your $DOOMDIR and profile init file are mutable, but in our case both of those live in the store. So the best case I'd expect is that doom sync does nothing, and although that is indeed what I'm getting, I suspect an error's getting swallowed. With our doom wrapper (which sets DOOMDIR and friends):

marienz@jian ~ > doom sync
- Using Emacs 29.4 @ /nix/store/jsxqmsj7v9xxni41xw87952xs29pvkvv-emacs-pgtk-29.4/bin/emacs
> Synchronizing "nix" profile...
  > Installing all packages for the first time (this may take a while)...
    - No packages need attention
  > Updating pinned packages...
    ✓ All 0 packages are up-to-date
  > Purging orphaned packages (for the emperor)...
    - Skipping builds
    - Skipping elpa packages
    - Skipping repos
    - Skipping regrafting
    - Skipping native bytecode
  > (Re)building profile in /nix/store/bbi1l0wy5219j2d76iv6ryw3l2n4in3d-doom-profile/profile/@/0/...
    > Deleting old init files...
✓ Finished in 1.23301s

Although it succeeded, strace shows it tried and failed to delete /nix/store/bbi1l0wy5219j2d76iv6ryw3l2n4in3d-doom-profile/profile/@/0/init.29.el, and with DEBUG=1 I get

  > (Re)building profile in /nix/store/bbi1l0wy5219j2d76iv6ryw3l2n4in3d-doom-profile/profile/@/0/...
    > Deleting old init files...
      - Deleting init.29.el...
      x There was an unexpected runtime error
        Message: File error
        Details: ("Removing old name" "Read-only file system" "/nix/store/bbi1l0wy5219j2d76iv6ryw3l2n4in3d-doom-profile/profile/@/0/init.29.el")

after which it fails. The first half says "0 packages" because it's reading the packages.el Unstraightened generated, which is telling Doom not to package-manage anything (because we're doing that for it). That means a profile init file generated by Doom is going to be incomplete (we extend that step...), but fortunately it tries to write the incomplete profile to the store, which it can't.

Next, doom/reload. This runs doom sync in an environment set up by doom--if-compile, which sets EMACS using invocation-directory and invocation-name, which is supposed to get it "the same Emacs we're currently running". We're starting Emacs through a few layers of wrapper, the last one of which is the one from emacsWithPackages, which execs the underlying emacs (with a bunch of env vars like EMACSLOADPATH set, but those get unset during startup).

So at that point we break in the way you identified: that inner Emacs doesn't have the packages we added, so we fall over (the first one we hit being straight).

So we can try to fix that bit, but the absolute best case scenario is "doom sync does nothing, without failing".

The next bit doom/reload does is "reload your init.el from doom-user-dir, then reload user-init-file (which is the profile init file)". Both of those will be the original ones from the store, so they should load. But they won't do what the Doom documentation tells you doom/reload is for (apply changes made to your packages.el or similar files).

Something similar applies to doom/upgrade, the only other user of doom--if-compile. It'll re-execute the current Emacs, not any new one.

[thecaralice]

Couple of things going on here...

Question: what do you want/expect doom/reload (SPC h r r) to do?

In vanilla Doom, you'd usually use this to test changes you made in your (mutable) $DOOMDIR / doom-user-dir. But in our case, all relevant paths point into the store: the best you're likely to get is "reload your original configuration" (the one you built the running doom-emacs with).

Is that what you want / expect? I don't want to spend too much time trying to make this not error out if the end result isn't going to be useful, and if that is what you want I'd be inclined to override doom/reload with something that just reloads config.el files (both for Doom modules and your private config).

Right, forgot to mention that. I have been working on a patch that would make doom use ~/.config/doom/ instead which would be a symlink to the store, so I could switch to a new home-manager configuration without having to restart Emacs, but stumbled upon this issue.

[marienz]

I have been working on a patch that would make doom use ~/.config/doom/ instead which would be a symlink to the store

Ah, I think I see.

...but although I'm not sure yet whether you're going for "reload config.el from doom-user-dir" or "reload the whole profile init file", I think in both cases you still won't actually want to run doom sync to make that work (so I'm inclined to leave the bug in "running doom CLI from Emacs" alone, because the environment setup in doom--if-compile looks a bit fiddly, and the only users of that function use it for doom sync, which I still think won't work).

During normal startup (as you may have had to track down already), loading ~/.config/doom/config.el is handled by some code generated around here and added to the profile init file (user-init-file). If you're doing what I think you're doing, we may end up with "edit your config, switch to new home-manager configuration (which includes instantiating our doom-profile derivation because doomdir is an input, our equivalent of doom sync), maybe reload ~/.config/doom/init.el through a symlink, load the profile init file through a symlink". Which would be doom/reload with the doom sync skipped.

I think that'd work if you're just changing config.el, I'm not sure to what extent it'd work if you're changing packages.el / init.el... Might need to try and see. And although you could reload config.el more directly, your home-manager switch is going to have to reinstantiate doom-profile anyway, so I suspect you might as well reload its profile init file and have that reload config.el for you.

...does that sound right, or am I misunderstanding what you're reloading / symlinking?

[thecaralice]

Yep, seems to match my experience. I later figured out this process is way more convoluted than what I thought initially. Still, a better way to iterate on the configuration would be nice to have.

[marienz]

Still, a better way to iterate on the configuration would be nice to have.

Not sure how much better we can really do... when tinkering with my own config, I'm usually editing just config.el or some adjacent file (in which case I directly re-evaluate that file while iterating) or I'm enabling a new Doom module or adding a new package (in which case I need to rebuild and restart anyway to get the new dependencies).

It's nice that these emacsWithPackages and Doom profile instantiations are completely isolated from each other (so your running instance doesn't pull in some undefined mix of dependency versions if you upgrade while it's running, and you can run the old and new one in parallel), but it does have the downside that they're completely isolated from each other (so you more or less have to start that new instance, reloading in place gets complicated)...