Update Doom, adjusting to initializating changes

Doom refactored its initialization, requiring some non-trivial changes
on our side to adapt:

- It now locates the profile init file by calling
  `doom-profile-init-file` instead of looking at `doom-profile-dir`,
  and loads the doom-user-dir's init.el from the profile init file
  instead of immediately before it. This means we need to move our
  advice of `doom-profile-init-file` into the profile loader to install
  it early enough (and rewrite it because its signature changed).
  Most of these changes are in commit
  8cafbe4408e7d6c68947a2066e5837379d0983c1 to Doom.
- The interface of profile generators changed (in commit
  b3aa41fd74241459bf1298b50bc3f39840236c1c to Doom).
- Some smaller changes to things like initialization order of straight.

These changes seem to be sufficient to make Unstraightened work again,
but they are subtle: I may have broken something.

This updates Doom to right before the next package bump. I tried
updating in stages, but ran into errors difficult to debug without
updating Doom to at least 0f556345fdfdc4e5bcc6bb023877bf9a0f203dae. That
pulls in almost all refactoring, so it made more sense to pull in
further fixes and the final refactor affecting me (the second signature
change to `doom-profile-init-file` in
87a024ee9070dbfaf39322e1bd45eac4c9597a26) as well.

HACKING.md

@@ -2,20 +2,14 @@
 
 ## Why [profiles](https://github.com/doomemacs/doomemacs/tree/master/profiles)?
 
-Because the profile loader runs early enough we can set `doom-profile-data-dir`
-(where the generated profile is stored and loaded from) outside `DOOMLOCALDIR`
-relatively cleanly. Not using the "global" profile is a largely unintended side
-effect, but the changes to other paths made seem largely reasonable so for now
-I'm sticking with it.
-
-After the profile loader, the next point we would get control is `doom-start.el`
-loading `init.el` from `doom-user-dir`. We currently point `doom-user-dir` into
-the store (see below): setting `doom-profile-data-dir` and `doom-profile-dir`
-from there (by prepending to the user's `init.el`) would probably also work.
-This seems a bit questionable because `doom-profile-dir` is set with `defconst`:
-its const-ness is not enforced but I'd prefer not to take advantage of that. I
-also have not checked if writing the profile would work out of the box with this
-approach.
+Because the profile loader runs early enough we can get Doom to load the profile
+init file from outside `DOOMLOCALDIR` relatively cleanly. Not using the "global"
+profile is a largely unintended side effect, but the changes to other paths made
+seem largely reasonable.
+
+We do not regain control in time after the loader without patching Doom: the
+first user code it runs is `init.el` from `doom-user-dir` (which we point into
+the store), but it loads that from the profile init file.
 
 Setting `profileName` to the empty string triggers a special case: we unset
 `DOOMPROFILE` from the profile loader. This feels like a hack, but it does get

build-helpers/build-doom-profile.sh

@@ -22,7 +22,6 @@ if ! [[ -e $out/doomdir/snippets ]]; then
 fi
 rm $out/doomdir/init.el
 substitute $initEl $out/doomdir/init.el \
-    --subst-var profileName \
     --subst-var-by userInit "$doomDir/init.el" \
     --subst-var-by straightBaseDir $out
 ln -sf $doomIntermediates/packages.el $out/doomdir/

build-helpers/build-profile

@@ -15,23 +15,23 @@
 ;; See the License for the specific language governing permissions and
 ;; limitations under the License.
 
-;; We skip Doom's normal install and initialization.
-(require 'straight)
-
 ;; Set from run!
 (defvar site-lisp)
 
 (defun generate-nix-autoloads ()
   "Pull in autoloads for non-ELPA emacsWithPackages dependencies."
-  (let (filenames)
-    (dolist (filename (directory-files site-lisp))
-      (when (string-suffix-p "-autoloads.el" filename)
-        (push (expand-file-name filename site-lisp) filenames)))
-    (doom-autoloads--scan filenames doom-autoloads-excluded-files 'literal)))
+  `((defun unstraightened--startup-nix-autoloads ()
+      (let ((load-in-progress t))
+        ,@(let (filenames)
+            (dolist (filename (directory-files site-lisp))
+              (when (string-suffix-p "-autoloads.el" filename)
+                (push (expand-file-name filename site-lisp) filenames)))
+            (doom-autoloads--scan
+             filenames doom-autoloads-excluded-files 'literal))))))
 
 (add-to-list
  'doom-profile-generators
- '("80-loaddefs-nix.auto.el" . generate-nix-autoloads))
+ '("80-loaddefs-nix.auto.el" generate-nix-autoloads unstraightened--startup-nix-autoloads))
 
 ;; Doom runs this with package.el activated, but suppresses activation during
 ;; normal startup. Store the side effects of activation in the profile to avoid
@@ -45,27 +45,31 @@
 
 (defun generate-unstraightened-autoloads ()
   "Like doom-profile--generate-package-autoloads but for package.el."
-  (doom-autoloads--scan
-   (mapcar (lambda (s)
-             (format "%s.el"
-                     (package--autoloads-file-name (package-get-descriptor s))))
-           ;; Packages are (currently...) pushed onto package-activated-list as
-           ;; they are activated. Reverse the list here so packages activated
-           ;; first get their autoloads loaded first.
-           ;;
-           ;; An example package that requires this is geiser-guile: it calls
-           ;; geiser-activate-implementation from autoloads, requiring geiser's
-           ;; autoloads are loaded first.
-           (nreverse
-            (seq-difference package-activated-list
-                            (mapcar #'intern-soft
-                                    doom-autoloads-excluded-packages))))
-   doom-autoloads-excluded-files
-   'literal))
+  `((defun unstraightened--startup-unstraightened-autoloads ()
+      (let ((load-in-progress t))
+        ,@(doom-autoloads--scan
+           (mapcar (lambda (s)
+                     (format "%s.el"
+                             (package--autoloads-file-name
+                              (package-get-descriptor s))))
+                   ;; Packages are (currently...) pushed onto
+                   ;; package-activated-list as they are activated. Reverse the
+                   ;; list here so packages activated first get their autoloads
+                   ;; loaded first.
+                   ;;
+                   ;; An example package that requires this is geiser-guile: it
+                   ;; calls geiser-activate-implementation from autoloads,
+                   ;; requiring geiser's autoloads are loaded first.
+                   (nreverse
+                    (seq-difference package-activated-list
+                                    (mapcar #'intern-soft
+                                            doom-autoloads-excluded-packages))))
+           doom-autoloads-excluded-files
+           'literal)))))
 
 (add-to-list
  'doom-profile-generators
- '("90-loaddefs-unstraightened.auto.el" . generate-unstraightened-autoloads))
+ '("90-loaddefs-unstraightened.auto.el" generate-unstraightened-autoloads unstraightened--startup-unstraightened-autoloads))
 
 (add-to-list 'doom-autoloads-cached-vars 'package-activated-list)
 
@@ -75,6 +79,12 @@
   ;; This also loads our generated profile's init.el. We need that both to get
   ;; the profile right and to load the advice to make Doom not install straight.
   (doom-modules-initialize)
+  ;; Now load straight (straight-prune-build-cache is not autoloaded).
+  ;;
+  ;; (Doom may call doom-initialize-packages / doom-initialize-core-packages
+  ;; from an after-load hook on straight, so we want to do this after we
+  ;; installed our advice of doom-initialize-core-packages above.)
+  (require 'straight)
   ;; Trigger a write of straight's build cache (which we write into the profile
   ;; and load again later).
   (straight-prune-build-cache)

build-helpers/build-profile-loader

@@ -24,7 +24,39 @@
          (new-profiles `((,profile-name
                           (user-emacs-directory . ,doom-emacs-dir)
                           (doom-profile-data-dir . ,profile-directory)
+                          (_ :eval
+;; HACK: install advice of doom-profile-init-file, because we do not gain
+;; control between here and Doom using this to find the profile init file
+;; (from which our own init.el is loaded).
+(define-advice doom-profile-init-file (:override (profile) unstraightened)
+  "Return unstraightened's profile init file.
+
+`doom-profile-init-file' locates the profile relative to `doom-data-dir', but
+nix-doom-emacs-unstraightened keeps its profile in a different location.
+Override `doom-profile-init-file' to confirm it is called to get the default
+or unstraightened profile (erroring out otherwise), then return the custom path.
+
+Returning the unstraightened profile if the default profile is
+requested makes `doom doctor' work."
+  (declare (side-effect-free t))
+  (let* ((my-profile ,(symbol-name profile-name))
+         ;; Open-code cl-destructuring-bind into (name . ref),
+         ;; to avoid loading cl in the profile loader.
+         (ref (if profile (doom-profile-key profile t) (cons nil nil)))
+         (name (pop ref)))
+    (unless (or (null name)
+                (and (not (string-empty-p my-profile))
+                     (string-equal name my-profile)))
+      (error "Accessing other profiles from Unstraightened is unsupported."))
+    (unless (or (null ref)
+                (string-equal "0" ref))
+      (error
+       "Accessing other profile versions from Unstraightened is unsupported."))
+    (file-name-concat doom-profile-dir
+                      (format "init.%d.%d.el"
+                              emacs-major-version emacs-minor-version))))
+)
                           ,@(when unset-profile '(("DOOMPROFILE" . nil)))))))
-    (doom-profiles-save new-profiles)))
+    (doom-profiles-write-load-file new-profiles)))
 
 (run! "build-profile-loader" (cdr (member "--" argv)))

init.el

@@ -31,31 +31,9 @@ it. Just skip it entirely."
   (require 'straight)  ;; straight-load-build-cache is not autoloaded.
   (straight--load-build-cache))
 
-(after! doom-packages
+(after! doom-straight
   (setq straight-base-dir "@straightBaseDir@"))
 
-(defadvice! unstraightened-profile-init-file (&optional profile-id version)
-  "Return unstraightened's profile init file.
-
-`doom-profile-init-file' locates the profile relative to `doom-data-dir', but
-nix-doom-emacs-unstraightened keeps its profile in a different location.
-Override `doom-profile-init-file' to confirm it is called to get the default
-or unstraightened profile (erroring out otherwise), then return the custom path.
-
-Returning the unstraightened profile if the default profile is
-requested makes `doom doctor' work."
-  :override #'doom-profile-init-file
-  (let ((my-profile "@profileName@"))
-    (unless (or (null profile-id)
-                (and (not (string-empty-p my-profile))
-                     (string-equal profile-id my-profile)))
-      (error "Accessing other profiles from Unstraightened is unsupported."))
-  (unless (null version)
-    (error
-     "Accessing other profile versions from Unstraightened is unsupported."))
-  (file-name-concat
-   doom-profile-dir (format "init.%d.elc" emacs-major-version))))
-
 ;; Doom adds a minor mode that makes flycheck-mode's emacs subprocess initialize
 ;; Doom. Extend this to set our profile dir before it does so.
 (setq-hook! +emacs-lisp--flycheck-non-package-mode