Setting NODE_OPTIONS in remoteEnv irretrievably breaks Codespace ⚠️

[Chuxel]

Discussed in https://github.com/orgs/devcontainers/discussions/14

^{Originally posted by davetapley December 29, 2022}
I inadvertently added a NODE_OPTIONS to remoteEnv instead of containerEnv, and it makes the Codespace irretrievably inaccessible as it get stuck in an infinite loop at "Setting up remote connection: Connecting to Codespace" (full output below).

Luckily I could use the Export Changes to a Branch feature to get my changes out, but still not ideal.

Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient V] connecting to relay link:sb://vsls-prod-ins-usw2-private-relay.servicebus.windows.net/3adea42d2c562dd9b87b56cd29af5fd560de--65c06416-8428-4d0e-8247-6e295e7702a4
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient V] creating node SSH...
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient V] authenticating...
[perf V] name:Creating the ssh workspace session duration:660.7617959976196
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient:SshWorkspaceSession V] subscribing to onDispose event
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient I] join started
[perf V] name:Ssh auth completion duration:294.8621253967285
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient E] auth failed
[perf V] name:join LiveShare session duration:957.0970239639282
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare E] connection failed: > object #0: Live Share client authentication failed.
  at /Users/dave/.vscode/extensions/workspace/client/dist/src/SshHelpers.js:96:31
  at /Users/dave/.vscode/extensions/node_modules/@vs/vs-ssh/sshClientSession.js:103:60
  at processTicksAndRejections (node:internal/process/task_queues:96:5)
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare I] connection status changed: connecting -> disconnected, error:> object #0: Live Share client authentication failed.
  at /Users/dave/.vscode/extensions/workspace/client/dist/src/SshHelpers.js:96:31
  at /Users/dave/.vscode/extensions/node_modules/@vs/vs-ssh/sshClientSession.js:103:60
  at processTicksAndRejections (node:internal/process/task_queues:96:5)
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare I] firing change event: > object #0: {
  "status": "disconnected",
  "previousStatus": "connecting"
}
[Client:extension:1439:liveshare:onConnectionChanged:2828D7F4DA679AAFEA40B1B64BDC0011A3EC I] connecting -> disconnected
[Client:extension:1439:liveshare:onConnectionChanged:2828D7F4DA679AAFEA40B1B64BDC0011A3EC I] disposing the connection
[Client:extension:1439:liveshare I] disposing 2828D7F4DA679AAFEA40B1B64BDC0011A3EC connection
[Client:extension:1439:liveshare:connection-record V] disposing reason: 'connection record disposed by ConnectionHub', disposing 2 objects
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare V] disposing reason: 'connection record disposed by ConnectionHub $1', disposing 3 objects
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient V] disposing reason: 'Disposed from LiveShareConnection', disposing 2 objects
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient I] stopCheckConnection reason:'disposed'
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient:SshWorkspaceSession V] disposing reason: 'Disposed from LiveShareConnection $1', disposing 2 objects
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:WorkspaceClient:SshWorkspaceSession:delayed-disposable V] Disposing reason:undefined
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare:delayed-disposable V] Disposing reason:undefined
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare V] signal rejected reason -> type:CancellationError props:{"stack":"Error: \n    at p.cancel (/Users/dave/.vscode/extensions/utils/dist/src/components/Signal.js:73:28)\n    at p.dispose (/Users/dave/.vscode/extensions/utils/dist/src/components/Signal.js:92:14)\n    at p.disposeObject (/Users/dave/.vscode/extensions/utils/dist/src/components/DisposableClass.js:30:28)\n    at /Users/dave/.vscode/extensions/utils/dist/src/components/DisposableClass.js:106:29\n    at Array.forEach (<anonymous>)\n    at p.dispose (/Users/dave/.vscode/extensions/utils/dist/src/components/DisposableClass.js:98:30)\n    at p.dispose (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connection/Connection.js:121:15)\n    at p.dispose (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connection/LiveShareConnection.js:57:15)\n    at p.disposeObject (/Users/dave/.vscode/extensions/utils/dist/src/components/DisposableClass.js:30:28)\n    at /Users/dave/.vscode/extensions/utils/dist/src/components/DisposableClass.js:106:29\n    at Array.forEach (<anonymous>)\n    at p.dispose (/Users/dave/.vscode/extensions/utils/dist/src/components/DisposableClass.js:98:30)\n    at h.disposeConnection (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connectionHub/ConnectionHub.js:230:30)\n    at h.<anonymous> (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connectionHub/ConnectionHub.js:209:18)\n    at Generator.next (<anonymous>)\n    at /Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connectionHub/ConnectionHub.js:8:71\n    at new Promise (<anonymous>)\n    at __awaiter (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connectionHub/ConnectionHub.js:4:12)\n    at h.onConnectionChanged (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connectionHub/ConnectionHub.js:195:16)\n    at g.invoke (/Users/dave/.vscode/extensions/node_modules/vscode-jsonrpc/lib/events.js:62:39)\n    at p.fire (/Users/dave/.vscode/extensions/node_modules/vscode-jsonrpc/lib/events.js:120:36)\n    at p.onConnectionChangedHandler (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connection/Connection.js:154:46)\n    at p.handleConnectionError (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connection/Connection.js:189:14)\n    at p.<anonymous> (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connection/LiveShareConnection.js:131:22)\n    at Generator.throw (<anonymous>)\n    at rejected (/Users/dave/.vscode/extensions/workspace/connection-hub/dist/src/connection/LiveShareConnection.js:6:57)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","errorType":"CancellationError"}
[Client:extension:1439:liveshare:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare E] connection failed: > object #0: Live Share client authentication failed.
Error: Live Share client authentication failed.
    at callback (/Users/dave/.vscode/extensions/workspace/client/dist/src/SshHelpers.js:96:31)
    at /Users/dave/.vscode/extensions/node_modules/@vs/vs-ssh/sshClientSession.js:103:60
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[Client:extension:1439:liveshare W] disposing connection "2828D7F4DA679AAFEA40B1B64BDC0011A3EC" due to an error> object #0: Live Share client authentication failed.
Error: Live Share client authentication failed.
    at callback (/Users/dave/.vscode/extensions/workspace/client/dist/src/SshHelpers.js:96:31)
    at /Users/dave/.vscode/extensions/node_modules/@vs/vs-ssh/sshClientSession.js:103:60
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[Client:extension:1439:liveshare I] disposing 2828D7F4DA679AAFEA40B1B64BDC0011A3EC connection
[Client:extension:1439:liveshare W] error during connection> object #0: Live Share client authentication failed.
Error: Live Share client authentication failed.
    at callback (/Users/dave/.vscode/extensions/workspace/client/dist/src/SshHelpers.js:96:31)
    at /Users/dave/.vscode/extensions/node_modules/@vs/vs-ssh/sshClientSession.js:103:60
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[perf V] name:Joining the workspace duration:299.3900308609009
[Client:extension:1439:liveshare-joint:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare-joint:liveshare E] connection failed: > object #0: Live Share client authentication failed.
Error: Live Share client authentication failed.
    at callback (/Users/dave/.vscode/extensions/workspace/client/dist/src/SshHelpers.js:96:31)
    at /Users/dave/.vscode/extensions/node_modules/@vs/vs-ssh/sshClientSession.js:103:60
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[Client:extension:1439:liveshare-joint:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare-joint:liveshare I] connection status changed: connecting -> disconnected, error:> object #0: Live Share client authentication failed.
Error: Live Share client authentication failed.
    at callback (/Users/dave/.vscode/extensions/workspace/client/dist/src/SshHelpers.js:96:31)
    at /Users/dave/.vscode/extensions/node_modules/@vs/vs-ssh/sshClientSession.js:103:60
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[Client:extension:1439:liveshare-joint:connection:2828D7F4DA679AAFEA40B1B64BDC0011A3EC:liveshare-joint:liveshare I] firing change event: > object #0: {
  "status": "disconnected",
  "previousStatus": "connecting"
}
```</div>

[Chuxel]

I assume this also affects the Dev Containers extension @chrmarti? Not sure if this is a CLI thing or Dev Containers/Codespaces.

[chrmarti]

With Dev Containers the extension host crashes when adding "NODE_OPTIONS": "--foo=123" to "remoteEnv". Removing that and rebuilding the container works. Using "NODE_OPTIONS": "" also gets the existing dev container back to working.

One difference to Codespaces is that for Codespaces we pass "remoteEnv" as env variables to the VS Code Server startup which should explain why Codespaces crashes more seriously.

Passing "NODE_OPTIONS" (as "remoteEnv" or "containerEnv") affects not only the user's Node.js, but also the VS Code Server, I recommend against using it. Maybe use it in your launch configs in launch.json instead.

@edgonmsft To improve recovery, Codespaces could offer using a recovery container. I thought this is already implemented, but maybe this case isn't covered yet?

[Chuxel]

Wouldn't it make sense to ensure that the crash doesn't happen? I believe that was what the original report was about - they went into a recovery container, but the reason why was not particularly clear.

[edgonmsft]

It would be interesting to figure out if there is a way to detect that we have this issue when connecting to the codespace or how to prevent it.

[chrmarti]

We could remove NODE_OPTIONS before passing the environment variables to the server / extension host.

[jkeech]

I noticed that Codespaces now get a NODE_OPTIONS=undefined environment variable set instead of unsetting the variable entirely. This is causing other issues for some Codespaces customers.