Emacs *shell*: the current directory is often not tracked; M-[Enter] may hang

[mmaechler]

M-x shell and *shell* buffers (several of them often) is what I use a lot in Emacs.
In shell-mode, M-[Enter] (aka M-RET (translated from <M-return>) runs the command dirs found in (shell-mode-map), which is an alias for shell-resync-dirs in shell.el)
makes sure that the stack of directories (I create and use via pushd , popd, etc) is kept in sync between the shell and emacs, notably emacs' own very important default-directory variable (which is buffer local in shell).

Since about 1-2 weeks now, this behaves erratically and slowish basically as soon as I have once opened and *.R script in a package .. which now automatically turns 'package development mode' for that buffer --- which is very convenient and nice.

What I cannot bear is that I basically can no longer work in may not so small stack of directories in *shell* where almost always only one of the directories is related to the package whose /R/*.R buffer I had opened previously.

Please tell us

1. how to revert via some customization ASAP --- it makes working with emacs very frustrating when it starts misbehaving badly... I typically need to close emacs and start a new one in order to not lose my last hairs..
2. do NOT make this behavior the default. It may be fine for people who only work with one R package at a time... but definitely not for me, and probably not for other multi-tasking R developers who like to do everything inside emacs.

[lionel-]

Since about 1-2 weeks now

Do you upgrade ESS often? Does 1-2 weeks correspond to such an ESS update? The last big change to the package mode was #470.

The parent commit is b439eb3, try to checkout this commit and see if that fixes it. Then see try 157a3a9 to see if that's indeed the cause of the bug. If not please bisect until you find the faulty commit.

how to revert via some customization ASAP

(setq ess-r-package-auto-activate nil)

do NOT make this behavior the default.

I think it would be premature to take this decision as we don't know what is going on. The package mode has been enabled by default since ESS 16.04.

[vspinu]

I have had a look at this and I am afraid the only place this could come from is the repeated calls to ess-r-package-name from the mode line updates. But that one is extremely light-weight due to the caching that we do and is very unlikely the cause of the slow down. There are not other calls to our functions during interaction with the shell.

If you have polymode on your machine there is a handy function pm-debug-trace-functions-by-regexp which allows you to trace all ess-r-package- functions at once.

First do M-: (pm-debug-trace-functions-by-regexp "^ess-r-package.*") then go to the shell buffer and open trace-output buffer alongside. All the calls to our functions will be displayed in that buffer.

I don't see any when I operate with shell buffer, so it must be your local setup which only you could investigate.

[vspinu]

This is the regexp tracing function:

(defun pm-debug-trace-background-1 (fn)
  (interactive (trace--read-args "Trace function in background: "))
  (unless (symbolp fn)
    (error "can trace symbols only"))
  (unless (get fn 'cl--class)
    (trace-function-background fn nil
                               '(lambda ()
                                  (format " [buf:%s pos:%s type:%s (%f)]"
                                          (current-buffer) (point)
                                          (get-text-property (point) :pm-span-type)
                                          (float-time))))))

(defun pm-debug-trace-functions-by-regexp (regexp)
  "Trace all functions whose name matched REGEXP."
  (cl-loop for sym being the symbols
           when (and (fboundp sym)
                     (not (eq sym 'pm-debug-trace-background-1)))
           when (string-match regexp (symbol-name sym))
           do (pm-debug-trace-background-1 sym)))

[mmaechler]

We are back to this (or have always been). Again, I find myself in a long running emacs with several shell buffers (each in comint), and M-[Enter] when in *shell* is first slow (timeout) and then prints command dir in the buffer itself and -- and that's the worst - then still does not track, i.e., does not make the default directory to the current (pwd) directory ....

Consequence: In a shell buffer I do

cd  <someplace>
C-x C-f <something> 

and emacs prompts me for files in a wrong directory/folder. This really inhibits efficiently working in Emacs for me.... and I think we agreed that anyway ESS should not at all ever take control over M-x shell buffers..

[jabranham]

I can't get this to reproduce. I'm doing:

emacs #which loads my whole config, including ESS
M-x shell
cd /somewhere
C-x C-f

and I get prompted for files in /somewhere.

As for the slowness, it might be helpful if you shared the output of the CPU profiler.

[mmaechler]

Thank you Alex, to reproduce you do need some R activity with ESS, probably even need open *.R files in a package and possibly even send code from there to the *R* buffer/process.

Loading the emacs lisp code of ESS is not a problem, it's the "package active" part of ESS which "takes over some / all (?) inferior *shell*s -- yes, I often have several *shell* buffers inside emacs and do all kind of things "simultaneously": Testing experimental code in R-devel (the development version of R). Running R CMD check --as-cran of one R package (with many tests, so taking many minutes). Running another sh loop over other things.. Many of the shells are often related to R.. but ESS package code should not assume it knows which package a shell "belongs to".

[lionel-]

Martin, it is not clear ESS is at fault. Please provide more information, e.g. profiler-start / profiler-report.

[mmaechler]

Fair point @lionel- . The problem appears (or aggravates) after a while of working only, so I have not found time yet to create ReprEx.. but next time (when I have time), I hope to remember profiler-start etc.

[mmaechler]

I'm still // again severely suffering from ESS slowing down my emacs *shell* interactions tremendeously.
Can I turn this off completely ... without turning off the Active part of package development ??
I need to be able to quickly get back to a sane state...

[lionel-]

I've noticed for a year or so that after using ESS for a while, Emacs stops accepting process outputs automatically. When this happens, the stdout pipes fill up and this blocks all processes sending outputs to Emacs, such as R or make. The output does go through after a while, though sporadically. The only way to get back to normal is to restart.

Is it possible we are observing the same thing? In that case it may not be related to ESS, I also observe this while coding in C with M-x compile. I didn't have the courage to debug this yet. It's possible it's a bug in Emacs itself.

[mmaechler]

Do you observe complete blocks? I usually get things going but still very slowly with C-g or C-c C-c etc... but still with lots of timeout. Yes, complete restart of emacs does help.
As it is very rare for me to have an emacs without an R running somewhere.. I cannot easily say if it's unrelated to ESS. Thank you, @lionel-, I'll try to keep your "piece of evidence" in mind.

[lionel-]

I think what I'm observing is that all processes block completely but when I do some actions like scrolling in a window it gets unblocked for a while. This may give the appearance of being slow.

How is your CPU usage when you're observing slowness? If it's low, we may be observing the same issue. If the usage is high, a profiler report might be useful.

[jabranham]

Does running the profiler produce any helpful information? Or setting toggle-debug-on-quit?

…

On Tue, Nov 26, 2019, 3:28 AM Lionel Henry ***@***.***> wrote: I *think* what I'm observing is that all processes block completely but when I do some actions like scrolling in a window it gets unblocked for a while. This may give the appearance of being slow. How is your CPU usage when you're observing slowness? If it's low, we may be observing the same issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#508?email_source=notifications&email_token=AC24ZCLTW7IP2IVLL7A7UDTQVTM27A5CNFSM4EWNGKNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFFFBRQ#issuecomment-558518470>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC24ZCMVZ4MLM3HW6OIGSKLQVTM27ANCNFSM4EWNGKNA> .

[vspinu]

FWIW, I don't see any of these. @lionel- how sure you are your problem is ESS related?

@mmaechler I think I asked you before, are any remotes involved in your workflow? Please try what @jabranham suggested 1) M-x profiler-start 2) do a few slow operations with shell 3) profiler-stop 4) profiler-report.

[lionel-]

@vspinu My assumption is that it's an Emacs problem, not an ESS problem.

[jabranham]

FWIW I think you need to switch steps 3 and 4 in the profiler in Emacs<27. I.e. profiler-report before profiler-stop.

…

On Wed, Dec 4, 2019, 7:44 AM Lionel Henry ***@***.***> wrote: @vspinu <https://github.com/vspinu> My assumption is that it's an Emacs problem, not an ESS problem. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#508?email_source=notifications&email_token=AC24ZCP4QRTIRD7J7HDWUBLQW6Q3HA5CNFSM4EWNGKNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF44DVI#issuecomment-561627605>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC24ZCIZSLCVUZUHHT35EPDQW6Q3HANCNFSM4EWNGKNA> .