avoid font-lock-fontify-buffer "Stack overflow in regexp matcher"

[tmarble]

When trying to cider-font-lock-as a very large string it is possible to get stack overflow like

Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
re-search-forward(...)
font-lock-fontify-keywords-region(1 46733 nil)
font-lock-default-fontify-region(1 46733 nil)
font-lock-fontify-region(1 46733 nil)
byte-code(...)
font-lock-default-fontify-buffer()
font-lock-fontify-buffer()
(progn (insert string) (set (make-local-variable (quote delay-mode-hooks)) t) (setq delayed-mode-hooks nil) (funcall mode) (font-lock-fontify-buffer) (buffer-string))
(unwind-protect (progn (insert string) (set (make-local-variable (quote delay-mode-hooks)) t) (setq delayed-mode-hooks nil) (funcall mode) (font-lock-fontify-buffer) (buffer-string)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
(save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert string) (set (make-local-variable (quote delay-mode-hooks)) t) (setq delayed-mode-hooks nil) (funcall mode) (font-lock-fontify-buffer) (buffer-string)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
(let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert string) (set (make-local-variable (quote delay-mode-hooks)) t) (setq delayed-mode-hooks nil) (funcall mode) (font-lock-fontify-buffer) (buffer-string)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
cider-font-lock-as(clojure-mode "... big string....")

This appears to tickle some limitations of large regexps:

• http://www.emacswiki.org/emacs/MultilineRegexp
• http://stackoverflow.com/questions/11269247/how-can-i-change-the-stack-size-available-for-emacs

Unsurprisingly the culprit is, again, Hacker News...
from https://github.com/swannodette/enlive-tutorial#your-first-scrape-with-enlive--hacker-news
loading src/tutorial/scrape1.clj in cider and running this will provoke the bug:

 ; CIDER 0.9.0snapshot (Java 1.8.0_40-internal, Clojure 1.7.0-beta1, nREPL 0.2.10)
tutorial.scrape1> (fetch-url *base-url*)

I have no idea what the "magic number" is for the biggest string that the regexp
can handle, but 32k seems to work... I propose this patch:

tmarble@ficelle 108 :) diff -u cider-util.el cider-util.el.tom
--- cider-util.el   2015-04-20 13:36:19.754803658 -0500
+++ cider-util.el.tom   2015-04-20 13:36:08.026963565 -0500
@@ -109,7 +109,12 @@
     (setq-local delay-mode-hooks t)
     (setq delayed-mode-hooks nil)
     (funcall mode)
-    (font-lock-fontify-buffer)
+    ;; avoid stack overflow
+    ;; http://www.emacswiki.org/emacs/MultilineRegexp
+    ;; http://stackoverflow.com/questions/11269247/how-can-i-change-the-stack-size-available-for-emacs
+    (if (< (buffer-size) 32768)
+        (font-lock-fontify-buffer)
+      (message "string too big to fontify..."))
     (buffer-string)))

 (defun cider-font-lock-region-as (mode beg end &optional buffer)
tmarble@ficelle 109 :( 

[bbatsov]

I'll have to think about this a bit. Your proposal seems fine (except for printing the message - I'd rather us simply not do anything then), but I wonder if this can be handled in a better manner - e.g. in batches or something.