Run both telemetry server and openmct visualizer Node.js applications through a single command

[nunoguedelha]

• We create a script for running both iCubTelemVizServer and openmctstaticServer in a single command (9f6e825)
• Further work is required for having an easier control of the process running the openmctstaticServer. For this purpose, we decided to run that process as a child process of iCubTelemVizServer (Run iCub telemetry server and openMCT visualizer in one line command: Iteration 1/2 #27 (comment), Run iCub telemetry server and openMCT visualizer in one line command: Iteration 1/2 #27 (comment))

Another way would be to run from the terminal only the iCubTelemetryServer, and then have the server itself run the openmctStaticServer as child process using the spawn command.
Actually this should be a better solution since installing and running the openmctStaticServer server separately was only due to the difference in the Node.js versions.

• Enhancement: we select the Node.js version in the openmctStaticServer start script instead of using a child process of iCubTelemVizServer (execSync command from the child_process package). Run both telemetry server and openmct visualizer Node.js applications through a single command #25 (comment), Run iCub telemetry server and openMCT visualizer in one line command: Iteration 2/2 #32 .

[nunoguedelha]

Edited the description. CC @traversaro

[nunoguedelha]

Run openmctstaticServer as a child process of iCubTelemVizServer

We need to perform the following operations at the end of the iCubTelemVizServer server startup:

cd ../openmctstaticServer
nvm use v14.17.0
npm start

Similarly to #13 (comment), we use 'spawn' method to launch the asynchronous child process executing the command npm start. This allow us to:

• stream in a controlled way the output of the command,
• run it in a new shell if necessary,
• close it along with iCubTelemVizServer with "CTRL+C" signal or if iCubTelemVizServer crashes.

This should be the most durable approach, as both openmctstaticServer and iCubTelemVizServer will be run by a single process as soon as iCubTelemVizServer can be installed with Node.js version 14.17.0.

Note

the command nvm use v14.17.0 can be run by a blocking child process (spawnSync) since the execution and output are short, as this command should only do a change of the used Node.js version. => 8bbbf2e.

[nunoguedelha]

Running "nvm use " with "spawnSync"

spawnSync cannot find the proper nvm command to run as it is not an executable, but rather a function.

Analysing...

[nunoguedelha]

Running "nvm use " with "spawnSync"|"execSync"

NVM is a shell script function defined in $HOME/.nvm/nvm.sh which can be run from a terminal under the following conditions:

• NVM is installed into $HOME/.nvm/.
• the script $HOME/.nvm/nvm.sh defining the NVM function is sourced when the terminal window is launched.

For such reason we can't just run nvm use v14.17.0 as a common shell command. Prior to doing so, we need to:

• Update the README in order to have NVM installation as a mandatory step instead of a recommendation (https://github.com/ami-iit/yarp-openmct/#server-installation).
• Get the $HOME environment variable from Node.js (https://nodejs.dev/learn/how-to-read-environment-variables-from-nodejs).
• Run the shell script $HOME/.nvm/nvm.sh through "spawnSync"|"execSync".
• Run the command "nvm use v14.17.0".

Note

"execSync" runs an underlying "spawnSync", but unlike the later, it spawns a shell and executes the command in that shell.

[nunoguedelha]

Run the nvm.sh script with "." or "sh" command

When we run an executable script as sh my-script.sh, the script is run in a new subshell, while when we run it using the "." (dot) command, as . my-script.sh, the script is run in the current shell context (https://www.shell-tips.com/bash/source-dot-command/). For having access to the function nvm() by running the nvm.sh script, without having to export it to the current shell context, we use the dot command.

"spawnSync" or "execSync" ?

• For the same reason, we would use "spawnSync" rather than "execSync", as the former runs the commands in the current shell context, while the later spawns a new child shell and executes the commands in that shell same.
• But if we run

  const { spawnSync } = require('child_process');
  spawnSync('.',[homeDir+'/.nvm/nvm.sh'],{timeout:3000});

  where homeDir is the $HOME path, we get the error:

  [ 'error: spawnSync . EACCES' ]

  This problem is not solved by simply changing the 'nvm.sh' file permissions (chmod 777), nor by setting the uid and gid options as {uid: process.getuid(),gid: process.getgid()}.
• We opt for "execSync" which does not encounter such problem.

The problem with "execSync" is that it spawns a sub-shell and runs the command from it, like "exec". The only difference with "exec" is that the "execSync" method wait for the process to terminate. As a consequence, in further function calls to "execSync", the new spawned shells' environments do not hold the previous definition of the "nvm" function, and we get the error "unknown command" or "ENOENT" if we try to run "nvm ls" or "nvm use v14.17.0".

Indeed, if we run twice the "execSync" as follows, the Node.js version switch through NVM is ineffective:

execSync(['. '+homeDir+'/.nvm/nvm.sh && nvm use v14.17.0 && nvm ls'],{timeout:3000});
console.log(stdout.toString()); ---> // returns v14.17.0
execSync(['. '+homeDir+'/.nvm/nvm.sh && nvm ls'],{timeout:3000});
console.log(stdout.toString()); ---> // returns v4.2.2

Workarounds to this issue:

• Concatenate the desired commands, running them through the same child process. For instance, for running the sequence . \<some-path\>/nvm.sh && command -v nvm && nvm ls, we do:

  const { execSync } = require('child_process');
  execSync(['. '+homeDir+'/.nvm/nvm.sh && command -v nvm && nvm ls'],{timeout:3000});

  and we get:

  $ nvm ls
  ->       v4.2.2
           v4.9.1
          v6.17.1
  

  But this approach is kind of cubersome and not flexible, as it requires to run also npm start trough the same child process. On top of that, the command "npm start" is runs the Open MCT server which should keep running after the

• Use "execFileSync" instead of "execSync", which does not run the command in a new shell but in the current shell context (option shell:false).

  execFileSync([homeDir+'/.nvm/nvm.sh'],[],{shell:false,timeout:3000});
  console.log(stdout.toString()); ---> // returns v14.17.0
  execFileSync('nvm',['ls'],{shell:false,timeout:3000});
  console.log(stdout.toString()); ---> // returns v4.2.2

• Export (export|declare -x) the defined functions AND/OR environment variables (*).

Only the first workaround was effective, but for avoiding to run all the commands in the same process, we just run the 2 commands:

. <NVM_DIR>/nvm.sh 1> /dev/null     // Load the NVM function script
nvm use v14.17.0 1> /dev/null '          // Set the NVM version
echo {\\"PATH\\":\\"$PATH\\", \\"NVM_INC\\":\\"$NVM_INC\\", \\"NVM_CD_FLAGS\\":\\"$NVM_CD_FLAGS\\", \\"NVM_BIN\\":\\"$NVM_BIN\\"}

The third command returns the NVM paths (updated by the command "nvm use vXX.XX.XX") in JSON formatted string, which can then be parsed by the parent process upon the completion of the commands and return of the execSync method.

(*) The execution of nvm use v14.17.0 affects the environment variables configuring NVM:

NVM_INC=$HOME/.nvm/versions/node/v14.17.0/include/node
NVM_BIN=$HOME/.nvm/versions/node/v14.17.0/bin
PATH includes the active $NVM_BIN.

Spawn for running 'npm start'

We usespawn for running 'npm start' under the path <yarp-openmct-path>/openmctStaticServer,after applying the environment variables NVM_INC, NVM_BIN and PATH which select the right Node.js version. We set the spawn options as follows:

• shell: we run the commands in a spawned shell, bash, since we are running a new Node.js process with a different environment configuration w.r.t. the parent process.
• env: parent environment with applied NVM_INC, NVM_BIN and PATH changes.
• cwd: current working directory where to run the process, <yarp-openmct-path>/openmctStaticServer.

Note

The other strong reason for using spawn instead of exec was that spawn streams the commands stdout to the parent process via a pipe, while exec buffers the output, which is then available once the command and child process are terminated.

I later realized that it could be possible to add the Node.js version selection in the start section of the package.json script, then just run "npm start" using spawn, without the initial step execSync for getting the NVM_BIN path:

  "scripts": {
    "start": ". $NVM_DIR/nvm.sh; nvm use v14.17.0; node server.js"
  },

This hasn't been tested for checking if it is compatible with spawn.

Refer to https://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options for more details.

[nunoguedelha]

Further observations on the code patterns used in the implementation

Paths and system dependent separators

We use the package path to define script files and working directory paths, for joining the path segments together while using the platform-specific separator, for portability purposes.

const path = require('path');

For instance, the nvm.sh path is defined by path.join(process.env.NVM_DIR, 'nvm.sh').

Commands sequence

A string with a sequence of commands can be defined in a multiple line assignment using the + operator:

const cmdString = 
    '<command_A>' +
    '&& <command_B>' +
    '&& <command_C>';

e.g., refer to https://www.designcise.com/web/tutorial/how-to-create-multi-line-strings-in-javascript.

Filtering command outputs

Typically, in a sequence of commands like the one meant for loading the NVM function script, setting the Node.js version and exporting the updated NVM environment variables, since we need to get in the spawn stdout stream only the last command output, we redirect the other commands stdout to /dev/null:

command_A 1> /dev/null && command_B 1> /dev/null && command_C

Creating a new environment object Y as a copy of X

There are three ways of coying an object, by using spread, Object.assign() or JSON (refer to https://www.javascripttutorial.net/object/3-ways-to-copy-objects-in-javascript/ and https://www.samanthaming.com/tidbits/70-3-ways-to-clone-objects/).

Both spread (...) and Object.assign() perform a shallow copy while the JSON methods carry a deep copy.

We apply the approach Object.assign() which is the officially supported one, to update the create from process.env the environment childProcEnv used with the child process running the OpenMCT Server. A shallow coy is acceptable because the structure process.env is flat anyways.

Other code references:

• https://nodejs.org/api
• https://www.w3schools.com/js
• https://www.tabnine.com/code/javascript
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/assign

[nunoguedelha]

Ciao @traversaro , for not making this issue too long and completing the 1rst iteration of the tool implementation, I've moved the analysis on the servers proper termination to #34.

Actually, the issues we are having when closing iCubTelemVizServer are independent from this pull request, as they will occur even if iCubTelemVizServer is running alone, and openmctStaticServer is never started, not even from the terminal.

I also checked that when we are running openmctStaticServer from iCubTelemVizServer through spawn, then stop iCubTelemVizServer from the terminal through <CTRL>+C and the main Node.js process closes properly (default <CTRL>+C event listener processing), the child process running openmctStaticServer also stops as expected, even without explicitly calling ChildProcess.kill().

I will update this issue's description accordingly as well as the PR.

[nunoguedelha]

Select the nodejs version in the 'openmctStaticServer' start script instead of using a child process of 'iCubTelemVizServer'.

I later realized that it could be possible to add the Node.js version selection in the start section of the package.json script, then just run "npm start" using spawn, without the initial step execSync for getting the NVM_BIN path:

  "scripts": {
    "start": ". $NVM_DIR/nvm.sh; nvm use v14.17.0; node server.js"
  },

This hasn't been tested for checking if it is compatible with spawn.

This actually works fine!

For this purpose, I did the following changes:

• Refactored OpenMctServerHandler.setNvmVersion for an eventual later use, but removed the call to this function in the main script, as well as the overwriting of the NVM environment variables in the spawn command running the Node.js child process.
• Added . ${NVM_DIR}/nvm.sh; nvm use <version>; in the start scripts of iCubTelemVizServer and openmctStaticServer. The first instruction defines the nvm function, while the second sets the Node.js version. In this case, the npm process running the start script can be launched by a 'spawn' command. Unlike running the command nvm use vX.X.X directly through spawn, using here an intermediate script ("start") avoids the EACCES problem encountered previously.

[nunoguedelha]

CC @S-Dafarra