IDE hangs from time to time

[RomanNikitenko]

Describe the bug

Within dogfooding session I faced with the problem that sometimes Che hangs
Looks like it starts from project tree items hang then IDE hangs in general, finally I get Connection is closed error or Paused before potential out-of-memory state (see screenshots section).
Please see video: https://youtu.be/3na80JGPMdw

Che version

I can reproduce it on dogfooding instance using fresh master of theia and che-theia, so my own images.
Also I tested it for devfile from happy tests for eclipse-che/che-theia#882.

Steps to reproduce

1. Try to expand tree items a few times

Screenshots

video: https://youtu.be/3na80JGPMdw

Environment

dogfooding instance

[RomanNikitenko]

Tested for https://che.openshift.io/

[RomanNikitenko]

Maybe related to upstream changes eclipse-theia/theia@fe4062e

[vitaliy-guliy]

Is it reproducible on local Che deployment?

[RomanNikitenko]

@vitaliy-guliy
I didn't check it

[RomanNikitenko]

I found that commit eclipse-theia/theia@329a3ef brought breaking changes for che.

[azatsarynnyy]

I'm labeling it as a blocker since it prevents us from consuming the latest upstream Theia commits with the required changes.

[tsmaeder]

@azatsarynnyy +1 on this being a blocker: let's not stay off theia/master too long.
Do we know how the plugin activation changes cause this behaviour?

[RomanNikitenko]

@tsmaeder

Do we know how the plugin activation changes cause this behaviour?

First we had suspicion that watcher related changes brought breaking changes for che, but after investigation I detected that the behavior was broken by eclipse-theia/theia@329a3ef
I'm investigating how the plugin activation changes cause this behaviour right now.

[RomanNikitenko]

@tsmaeder
about your question
looks like plugin activation process like endless loop now:

First I see tons of logs:

2020-10-22 08:40:46.301 root ERROR [hosted-plugin: 46] activatePlugin:  editorconfig.editorconfig 
2020-10-22 08:40:46.390 root ERROR [hosted-plugin: 46] activatePlugin:  editorconfig.editorconfig 
2020-10-22 08:40:46.476 root ERROR [hosted-plugin: 46] activatePlugin:  editorconfig.editorconfig 
2020-10-22 08:40:46.800 root ERROR [hosted-plugin: 46] activatePlugin:  editorconfig.editorconfig 
2020-10-22 08:40:47.262 root ERROR [hosted-plugin: 46] activatePlugin:  editorconfig.editorconfig 

then

2020-10-22 08:40:47.562 root INFO [hosted-plugin: 46] <--- Last few GCs --->
e [46:0x55d9dccc1020]   802260 ms: Mark-sweep 383.2 (387.9) -> 380.7 (386.7) MB, 65.9 / 0.0 ms  (+ 279.2 ms in 404 steps since start of marking, biggest step 7.0 ms, walltime since start of marking 357 ms) (average mu = 0.158, current mu = 0.035) finalize i[46:0x55d9dccc1020]   802774 ms: Mark-sweep 381.3 (386.9) -> 380.0 (385.2) MB, 4.9 / 0.0 ms  (+ 224.1 ms in 446 steps since start of marking, biggest step 7.6 ms, walltime since start of marking 366 ms) (average mu = 0.355, current mu = 0.555) finalize in

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x0b414b3c08d1 <JSObject>
    0: builtin exit frame: writeUtf8String(this=0x31ba51c8b661 <Pipe map = 0xfd504559399>,0x1bdede0004b1 <undefined>,0x24d72d61bfc9 <Very long string[17741]>,0x24d72d6198a1 <WriteWrap map = 0x3119db13c079>,0x31ba51c8b661 <Pipe map = 0xfd504559399>)

    1: _send [0x86e90326b59] [internal/child_process.js:~688] [pc=0x27ca8e042f95](this=0x2448f643fd41 <process map = 0x3119db135fe9>,0x24d72d6197d9 ... 
2020-10-22 08:40:47.562 root ERROR [hosted-plugin: 46] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 
2020-10-22 08:40:48.395 root ERROR [hosted-plugin: 46] IPC exited, with signal: SIGABRT, and exit code: null 

Pay attention on JavaScript heap out of memory ^^

then tons of logs like:

2020-10-22 08:40:48.407 root ERROR Error from plugin host: Channel closed 
2020-10-22 08:40:48.500 root ERROR Error from plugin host: Channel closed 
2020-10-22 08:40:48.591 root ERROR Error from plugin host: Channel closed 
2020-10-22 08:40:49.166 root ERROR Error from plugin host: Channel closed 

then

2020-10-22 09:11:54.791 root ERROR Error: Connection got disposed.
    at Object.dispose (/home/theia/node_modules/vscode-jsonrpc/lib/main.js:904:25)
    at /home/theia/node_modules/vscode-ws-jsonrpc/lib/socket/connection.js:14:41
    at CallbackList.invoke (/home/theia/node_modules/vscode-jsonrpc/lib/events.js:62:39)
    at Emitter.fire (/home/theia/node_modules/vscode-jsonrpc/lib/events.js:121:36)
    at closeHandler (/home/theia/node_modules/vscode-jsonrpc/lib/main.js:240:26)
    at CallbackList.invoke (/home/theia/node_modules/vscode-jsonrpc/lib/events.js:62:39)
    at Emitter.fire (/home/theia/node_modules/vscode-jsonrpc/lib/events.js:121:36)
    at WebSocketMessageReader.fireClose (/home/theia/node_modules/vscode-jsonrpc/lib/messageReader.js:111:27)
    at WebSocketMessageReader.fireClose (/home/theia/node_modules/vscode-ws-jsonrpc/lib/socket/reader.js:67:19)
    at /home/theia/node_modules/vscode-ws-jsonrpc/lib/socket/reader.js:24:18 

and

[tsmaeder]

That makes sense: plugin activation should be a no-op after the first time, the referenced commit broke that behaviour, obviously.

[azatsarynnyy]

fixed by eclipse-che/che-theia#891