Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

byte compile in child process #657

Merged
merged 2 commits into from
Dec 31, 2020
Merged

byte compile in child process #657

merged 2 commits into from
Dec 31, 2020

Conversation

progfolio
Copy link
Contributor

Work in progress using make-process to byte-compile packages in a child process.
Haven't dealt with the fact that we've made byte-compilation asynchronous.
For example, the :post-build commands should probably be executed in a callback after compilation.

@progfolio progfolio marked this pull request as draft December 21, 2020 04:19
@progfolio progfolio mentioned this pull request Dec 21, 2020
Closed
@raxod502
Copy link
Member

Making byte-compilation asynchronous sketches me out a fair amount. I think we might want to keep it synchronous at least by default. Otherwise, it'll be happening in parallel with the rest of Emacs startup, which could involve loading the very package being byte-compiled. I can't point to a specific race condition that would happen with this kind of scenario, but it feels very scary.

Converting fully general asynchronous process invocations into synchronous calls is pretty hard (see my struggles in #334), but doing it for ones that don't need to read from stdin should be easy.

@progfolio
Copy link
Contributor Author

Sounds good. Top priority is fixing #624. We can always expand on the idea later. I'll rework this so it's synchronous.

@progfolio progfolio force-pushed the feat/byte-compilation-76 branch 4 times, most recently from 4409c4d to 2f2ebbc Compare December 22, 2020 22:15
@progfolio progfolio marked this pull request as ready for review December 23, 2020 01:35
@progfolio progfolio force-pushed the feat/byte-compilation-76 branch 2 times, most recently from 2118623 to 0219a4f Compare December 23, 2020 22:32
@progfolio
Byte compile in child process
27839e2 
Separates byte-compilation from state of running Emacs.

See: radian-software#76, radian-software#624
@progfolio
Copy link
Contributor Author

progfolio commented Dec 28, 2020 •
edited
Loading

The following test case seems to work properly.
The files are byte-compiled and Org is not loaded during the process:

Test Case 
(straight-bug-report
  :user-dir "byte-compile-child-process.straight"
  :pre-bootstrap 
  (setq straight-repository-user "progfolio" straight-repository-branch "feat/byte-compilation-76")
  :post-bootstrap 
  (eval-after-load "org"
    '(message "FAILURE: Org loaded"))
  (straight-use-package
   `(org :type git :repo "https://code.orgmode.org/bzg/org-mode.git" :no-autoloads t :local-repo "org" :files
         (:defaults "contrib/lisp/*.elc" "lisp/*.elc")
         :build ,(let
                     ((make
                       (if
                           (eq system-type 'berkeley-unix)
                           "gmake" "make"))
                      (emacs
                       (concat "EMACS=" invocation-directory invocation-name)))
                   `((,make "autoloads" ,emacs)))))
  (straight-use-package 'org-journal)
  (with-current-buffer
      (get-buffer-create straight-byte-compilation-buffer)
    (message "%s"
             (buffer-string))))
  • Test run at: 2020-12-27 19:31:46
  • system-type: gnu/linux
  • straight-version: prerelease (HEAD -> develop, origin/develop) 1777741 2020-12-22
  • emacs-version: GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.17.4, Xaw3d scroll bars) of 2020-12-23
Output 
Bootstrapping straight.el...
Bootstrapping straight.el...done
Rebuilding all packages due to build cache schema change
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...
Looking for emacsmirror-mirror recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> feat/byte-compilation-76, origin/feat/byte-compilation-76) 27839e2 2020-12-23
Cloning org...
Cloning org...done
Building org...
Building org...done
Cloning org-journal...
Cloning org-journal...done
Building org-journal...
Building org-journal...done

Checking /tmp/byte-compile-child-process.straight/straight/build/straight/...
Compiling /tmp/byte-compile-child-process.straight/straight/build/straight/straight-autoloads.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/straight/straight-x.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/straight/straight.el...
Done (Total of 2 files compiled, 1 skipped)
Checking /tmp/byte-compile-child-process.straight/straight/build/org/...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-C.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-J.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-R.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-abc.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-asymptote.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-awk.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-calc.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-clojure.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-comint.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-coq.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-core.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-css.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-ditaa.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-dot.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-ebnf.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-emacs-lisp.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-eshell.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-eval.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-exp.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-forth.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-fortran.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-gnuplot.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-groovy.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-haskell.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-hledger.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-io.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-java.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-js.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-latex.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-ledger.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-lilypond.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-lisp.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-lob.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-lua.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-makefile.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-matlab.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-maxima.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-mscgen.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-ocaml.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-octave.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-org.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-perl.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-picolisp.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-plantuml.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-processing.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-python.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-ref.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-ruby.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-sass.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-scheme.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-screen.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-sed.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-shell.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-shen.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-sql.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-sqlite.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-stan.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-table.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-tangle.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob-vala.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ob.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-bbdb.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-bibtex.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-docview.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-eshell.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-eww.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-gnus.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-info.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-irc.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-mhe.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-rmail.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol-w3m.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ol.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-agenda.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-archive.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-attach-git.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-attach.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-capture.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-clock.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-colview.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-compat.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-crypt.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-ctags.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-datetree.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-duration.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-element.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-entities.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-faces.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-feed.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-footnote.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-goto.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-habit.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-id.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-indent.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-inlinetask.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-install.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-keys.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-lint.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-list.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-loaddefs.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-macro.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-macs.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-mobile.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-mouse.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-num.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-pcomplete.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-plot.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-protocol.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-refile.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-src.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-table.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-tempo.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-timer.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org-version.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/org.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-ascii.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-beamer.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-html.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-icalendar.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-latex.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-man.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-md.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-odt.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-org.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-publish.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox-texinfo.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org/ox.el...
Done (Total of 124 files compiled, 3 skipped)
Checking /tmp/byte-compile-child-process.straight/straight/build/org-journal/...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org-journal/org-journal-autoloads.el...
Compiling /tmp/byte-compile-child-process.straight/straight/build/org-journal/org-journal.el...
Done (Total of 1 file compiled, 1 skipped)

raxod502
raxod502 reviewed Dec 31, 2020
View reviewed changes
straight.el Outdated Show resolved Hide resolved
@raxod502
Copy link
Member

LGTM!

@progfolio @raxod502
Update straight.el
c715db0 
Co-authored-by: Radon Rosborough <[email protected]>
@progfolio progfolio merged commit 48296ba into radian-software:develop Dec 31, 2020
@progfolio progfolio deleted the feat/byte-compilation-76 branch December 31, 2020 20:19
@progfolio progfolio mentioned this pull request Mar 9, 2021
Closed
@raxod502 raxod502 mentioned this pull request Sep 3, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raxod502 raxod502 raxod502 left review comments

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@progfolio @raxod502