Parallelize referred-syms-by-file&fullname
#320
Conversation
| @@ -184,7 +188,10 @@ | |||
| (defn find-in-project | |||
| "Return the files in the project satisfying (pred ^File file)." | |||
| [pred] | |||
| (-> find-in-dir (partial pred) (mapcat (dirs-on-classpath)) distinct)) | |||
| (->> (dirs-on-classpath) | |||
| (pmap (partial find-in-dir pred)) | |||
There was a problem hiding this comment.
I chose pmap for simplicity, it's better than some tend to say
(because pmap won't create one thread per item... it takes the CPU count in account)
...it would be inefficient for very small inputs, but it's safe to say that most projects have multiple dirs in the classpath (especially in monorepos), plenty of namespaces, etc.
There was a problem hiding this comment.
Good choice! The small workloads that will get slightly slower will still be more than fast enough in the face of some fork/join overhead, I think :)
| @@ -88,10 +88,10 @@ | |||
| {:clj (->> (core/find-in-project (util/with-suppressed-errors | |||
| (some-fn core/clj-file? core/cljc-file?) | |||
| ignore-errors?)) | |||
| (map (juxt identity (partial get-libspec-from-file-with-caching :clj))) | |||
| (pmap (juxt identity (partial get-libspec-from-file-with-caching :clj))) | |||
There was a problem hiding this comment.
I'd add a note about the use of pmap, as people tend to "optimize" it by removing it. :D
CHANGELOG.md
Outdated
| @@ -3,6 +3,8 @@ | |||
| ## Unreleased | |||
|
|
|||
| #### Changes | |||
| * [(Part of #230)](https://github.com/clojure-emacs/refactor-nrepl/issues/230) Parallelize various fuctionality | |||
There was a problem hiding this comment.
One blank line is missing. And a :.
| @@ -184,7 +188,10 @@ | |||
| (defn find-in-project | |||
| "Return the files in the project satisfying (pred ^File file)." | |||
| [pred] | |||
| (-> find-in-dir (partial pred) (mapcat (dirs-on-classpath)) distinct)) | |||
| (->> (dirs-on-classpath) | |||
| (pmap (partial find-in-dir pred)) | |||
There was a problem hiding this comment.
Good choice! The small workloads that will get slightly slower will still be more than fast enough in the face of some fork/join overhead, I think :)
vemv
force-pushed
the
230--parallelize-namespace-aliases
branch
from
932cd34 to
02bc6d3
Compare
|
🍻 |
Part of #230
Parallelizes
referred-syms-by-file&fullnameandcore/functions used byreferred-syms-by-file&fullname.(the
core/parallelization will be only noticable in large projects)One can verify the improvement with the following repl input over refactor-nrepl itself:
...It's clearly better than
masterperformance for both the non-cached and cached (i.e. subsequent invocations) cases.(the numbers of course will depend on your CPU count)