Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nx:run-commands hangs when running multiple commands sequentially #21779

Closed
3 of 4 tasks
ejasarevic opened this issue Feb 12, 2024 · 23 comments · Fixed by #22294
Closed
3 of 4 tasks

nx:run-commands hangs when running multiple commands sequentially #21779

ejasarevic opened this issue Feb 12, 2024 · 23 comments · Fixed by #22294
Assignees

Comments

@ejasarevic
Copy link

ejasarevic commented Feb 12, 2024

Current Behavior

Having a simple target inside project.json -> targets

"test": {
      "executor": "nx:run-commands",
      "options": {
        "commands": [
          "echo 'test 1'",
          "echo 'test 2'"
        ],
        "parallel": false
      }
    },

It only prints 'test 1' and hangs indefinitely. It is not possible to stop the process.
Tested it with NX 17.3.2 and it works as expected with that version.

Expected Behavior

Both commands are executed sequentially (non parallel).
Output is:
test 1
test 2

GitHub Repo

No response

Steps to Reproduce

  1. Create new NX App with NX 18.x
  2. Add a new target to project.json with executor "executor": "nx:run-commands",
  3. Set "parallel: false"
  4. Add any 2 commands inside "options" -> "commands" array

Nx Report

Node   : 18.18.0                                         
OS     : win32-x64                                       
npm    : 9.8.1

   nx                 : 18.0.3
   @nx/js             : 18.0.3
   @nx/jest           : 18.0.3
   @nx/linter         : 18.0.3
   @nx/eslint         : 18.0.3
   @nx/workspace      : 18.0.3
   @nx/devkit         : 18.0.3
   @nx/eslint-plugin  : 18.0.3
   @nx/next           : 18.0.3
   @nx/playwright     : 18.0.3
   @nx/react          : 18.0.3
   @nrwl/tao          : 18.0.3
   @nx/web            : 18.0.3
   @nx/webpack        : 18.0.3
   typescript         : 5.3.3

Failure Logs

No response

Package Manager Version

9.8.1

Operating System

  • macOS
  • Linux
  • Windows
  • Other (Please specify)

Additional Information

No response

@danhobbs75
Copy link

danhobbs75 commented Feb 14, 2024

I am experiencing this with Nx 18.0.4. I tried a workaround where I split a prebuild step into two where each has a single command instead of an array of commands, and now my build dependsOn ["prebuild1","prebuild2"]. The workaround is fine for me in a simple case of running two commands. HOWEVER, the build step then hangs. I can run each individually but obviously this is not ideal.

@kwextpb
Copy link

kwextpb commented Feb 14, 2024

Not really a workaround and it scales horribly, cause if you had let's say 7 commands you would need to add 7 targets with a lot of syntax sugar. As a workaround I would suggest for now downgrade nx to 17.

@danhobbs75
Copy link

Not really a workaround and it scales horribly, cause if you had let's say 7 commands you would need to add 7 targets with a lot of syntax sugar. As a workaround I would suggest for now downgrade nx to 17.

You are quite right that it doesn't scale, and I only mention it to back up that I've seen the same behaviour, and in case anyone else only has two commands and is happy to wait for a fix by doing what I'm doing (in fact, only one command with a chained dependsOn seems to have the same problem as I see you've mentioned elsewhere - behaviour I also see).

@Bonscho
Copy link

Bonscho commented Feb 16, 2024

Can confirm the issue. Are there news regarding a workaround or solution? Otherwise we would need to downgrade ;(

@rockse
Copy link
Contributor

rockse commented Feb 16, 2024

We are facing the same issue

@Cammisuli
Copy link
Member

I'll look into this as soon as possible. In the meantime, can you confirm that things work as expected when setting the env var NX_NATIVE_COMMAND_RUNNER=false?

@danhobbs75
Copy link

I'll look into this as soon as possible. In the meantime, can you confirm that things work as expected when setting the env var NX_NATIVE_COMMAND_RUNNER=false?

Same result for me: hangs when executing my original prebuild task which has two commands.

@Ionaru
Copy link

Ionaru commented Feb 17, 2024

@Cammisuli

Using NX_NATIVE_COMMAND_RUNNER=false does fix the issue for me with the simpler task syntax.

  "targets": {
    "some-task": {
      "command": "echo 'Command 1'"
    },
  ...

And it also seems to work when I use the nx:run-commands syntax.

  "targets": {
    "some-task": {
      "executor": "nx:run-commands",
      "options": {
        "commands": [
          "echo 'Command 1'",
          "echo 'Command 2'"
        ],
        "parallel": false
      }
    },
  ...

@Bonscho
Copy link

Bonscho commented Feb 19, 2024

I can confirm that setting the NX_NATIVE_COMMAND_RUNNER env to false solves the problem for us with the following target setup.

"copy-assets": {
	"executor": "nx:run-commands",
	"options": {
		"commands": [
			{
				"command": "some command",
				"forwardAllArgs": false
			},
			{
				"command": "some other command",
				"forwardAllArgs": false
			}
		],
		"parallel": false
	}
}

and

"build": {
	"executor": "@nx/webpack:webpack",
	"outputs": ["{options.outputPath}"],
	"dependsOn": [
		{
			"target": "copy-assets"
		},
		
	], 
	
}

@danhobbs75
Copy link

danhobbs75 commented Feb 19, 2024

I'll look into this as soon as possible. In the meantime, can you confirm that things work as expected when setting the env var NX_NATIVE_COMMAND_RUNNER=false?

Same result for me: hangs when executing my original prebuild task which has two commands.

Actually, I'm going to rephrase my answer having seen that @Bonscho had success. The task is running but I get no output except the result of the Boolean expression to run two commands in Powershell:

($env:NX_NATIVE_COMMAND_RUNNER = "false") -and (nx prebuild)

The above outputs true, but I can tell that the prebuild task HAS been executed but I can't see the output

Then I get this result when I try to run one of two commands on its own via a separate Nx run-commands target (nx prebuild1 - where I have the first of the two commands on its own):

 >  NX   Nx Cloud Problems

   Error: write EPIPE

Messages from the log:
[NX Daemon Server] - 2024-02-19T09:05:07.068Z - Done sending EndRun request
[NX Daemon Server] - 2024-02-19T10:00:47.982Z - Established a connection. Number of open connections: 1
[NX Daemon Server] - 2024-02-19T10:00:47.982Z - Closed a connection. Number of open connections: 0
[NX Daemon Server] - 2024-02-19T10:00:47.983Z - Established a connection. Number of open connections: 1
[NX Daemon Server] - 2024-02-19T10:00:47.988Z - [REQUEST]: Client Request for Project Graph Received
[NX Daemon Server] - 2024-02-19T10:00:47.988Z - [REQUEST]: Responding to the client. project-graph
[NX Daemon Server] - 2024-02-19T10:00:47.991Z - Time taken for 'total for creating and serializing project graph' 0.31610000133514404ms
[NX Daemon Server] - 2024-02-19T10:00:47.992Z - Done responding to the client project-graph
[NX Daemon Server] - 2024-02-19T10:00:47.992Z - Handled REQUEST_PROJECT_GRAPH. Handling time: 0. Response time: 4.
[NX Daemon Server] - 2024-02-19T10:00:48.092Z - [REQUEST]: Responding to the client. handleHashTasks
[NX Daemon Server] - 2024-02-19T10:00:48.093Z - Done responding to the client handleHashTasks
[NX Daemon Server] - 2024-02-19T10:00:48.093Z - Handled HASH_TASKS. Handling time: 31. Response time: 1.
[NX Daemon Server] - 2024-02-19T10:00:50.168Z - [WATCHER]: Processing file changes in outputs
[NX Daemon Server] - 2024-02-19T10:00:50.170Z - [REQUEST]: Responding to the client. recordOutputsHash
[NX Daemon Server] - 2024-02-19T10:00:50.170Z - [WATCHER]: Stopping the watcher for D:\GitHub\bb (sources)
[NX Daemon Server] - 2024-02-19T10:00:50.170Z - Done responding to the client recordOutputsHash
[NX Daemon Server] - 2024-02-19T10:00:50.170Z - Handled RECORD_OUTPUTS_HASH. Handling time: 0. Response time: 0.
[NX Daemon Server] - 2024-02-19T10:00:50.170Z - [WATCHER]: Stopping the watcher for D:\GitHub\bb (outputs)
[NX Daemon Server] - 2024-02-19T10:00:50.170Z - Server stopped because: "Lock file changed"



More information: D:\GitHub\bb\.nx\cache\d\daemon.log

So something is not happy. I get this error report and log snippet every time I run an Nx task subsequently. The prebuild1 task which has got one command does always run successfully, and I can see its output in the console window, but I also get that bonus error report.

UPDATE:

I changed the command I'm running to this and now I DO see the output:

$env:NX_NATIVE_COMMAND_RUNNER = "false" ; nx prebuild  

I'm not a Powershell expert, it's fair to say. :-)

Still getting the error report if I run the prebuild1 task with the single command.

@bubblegumsoldier
Copy link

@Cammisuli

Using NX_NATIVE_COMMAND_RUNNER=false does fix the issue for me with the simpler task syntax.

  "targets": {
    "some-task": {
      "command": "echo 'Command 1'"
    },
  ...

And it also seems to work when I use the nx:run-commands syntax.

  "targets": {
    "some-task": {
      "executor": "nx:run-commands",
      "options": {
        "commands": [
          "echo 'Command 1'",
          "echo 'Command 2'"
        ],
        "parallel": false
      }
    },
  ...

I can confirm this works for me.

However I was trying to be a smartypants and did this

        "start": {
            "executor": "nx:run-commands",
            "dependsOn": ["build"],
            "options": {
                "commands": ["my-command"],
                "env": {
                    "NX_NATIVE_COMMAND_RUNNER": "false"
                },
                "parallel": false
            }
        }

But that did not work

@Cammisuli
Copy link
Member

Cammisuli commented Feb 22, 2024

Hmm.. I tried giving this a shot locally with doing echos and it looks like things are completing properly for me. Whats another command where you all noticed this was hanging with?

Here's a screenshot of my local run:
image

@Ionaru
Copy link

Ionaru commented Feb 23, 2024

@Cammisuli You are running the target directly. The problem (at least, for me), exists when you have your temp target in a dependsOn of another target.

Add "dependsOn": ["temp"] in your build target and run parallel-test:build.

@danhobbs75
Copy link

Hmm.. I tried giving this a shot locally with doing echos and it looks like things are completing properly for me. Whats another command where you all noticed this was hanging with?

My prebuild target looks like this:

    "prebuild": {
      "executor": "nx:run-commands",
      "outputs": [],
      "options": {
        "commands": [
          "echo 'prebuild'",
          "npm --no-git-tag-version version patch",
          "echo 'version patch done'",
          "node extract-version.js",
          "echo 'extract-version.js done'"
        ],
        "parallel": false
      }

My build dependsOn it.

But also I separated them so this will run fine if targeted directly:

    "prebuild1": {
      "executor": "nx:run-commands",
      "outputs": [],
      "options": {
        "command": "npm --no-git-tag-version version patch"
      }
    },

...but the moment I make the build depend on that prebuild1 target, it also hangs.

I hope that helps you track it down. Sounds like an annoying "works for me" one which is difficult to replicate. :-(

@Cammisuli
Copy link
Member

Ok I’ll check again with that.

@JakeGinnivan
Copy link
Contributor

My colleague had issues with some WSL process running away.

I wonder if it's something to do with using Terminal (non-wsl) then behind the scenes NX is shelling commands which are running in WSL and something between the two environments is going bad.

When the issue happened for him, this process went runaway.

image

It kept growing in memory usage and was sustaining ~100MB/s disk usage until his machine grinds to a halt and needs a hard reset.

@Inkdpixels
Copy link

Just chiming in since we just experienced quite a similar issue but I have no additional information wether it is a separate one. We also have targets with sequential nx:run-commands which fail to execute / hang after 1-2 commands.

Funnily enough, once we add --output-style=stream, the commands succeed.

So TL;DR

npx nx run <project>:<target> --verbose                         # fails
npx nx run <project>:<target> --verbose --output-style=stream   # succeeds

Also interestingly, I could not observe the failures when invoking the target through nx run-many. Maybe this helps anyone?

Node   : 20.10.0
OS     : win32-x64
npm    : 10.2.3

nx                 : 18.0.5
@nx/js             : 18.0.5
@nx/jest           : 18.0.5
@nx/linter         : 18.0.5
@nx/eslint         : 18.0.5
@nx/workspace      : 18.0.5
@nx/devkit         : 18.0.5
@nx/eslint-plugin  : 18.0.5
@nx/express        : 18.0.5
@nx/node           : 18.0.5
@nx/plugin         : 18.0.5
@nx/react          : 18.0.5
@nrwl/tao          : 18.0.5
@nx/web            : 18.0.5
@nx/webpack        : 18.0.5
typescript         : 5.3.3

@atamike
Copy link

atamike commented Mar 6, 2024

I ran today into the same problem. We have updated from NX17 to NX18 and the problem occurs. My collegues which are using mostly Mac are fine, the problem is only for me and those who are using Windows.

@BenPag
Copy link

BenPag commented Mar 8, 2024

One other fix, that works for me, is the following and after that everything works fine under windows.

Adding NX_NATIVE_COMMAND_RUNNER=false to the .env file at workspace root. The .env file was introduced with NX18.

@danhobbs75
Copy link

One other fix, that works for me, is the following and after that everything works fine under windows.

Adding NX_NATIVE_COMMAND_RUNNER=false to the .env file at workspace root. The .env file was introduced with NX18.

This is a great workaround and also works for me (running on Windows 11). Thanks for sharing this one.

@MoeedthePessimist
Copy link

I am having the same issue with building my react frontend. The build commands hangs at one point after builiding all the libraries. I have also tried using the command after adding NX_NATIVE_COMMAND_RUNNER=false to the root .env.

I have generated a ticket but haven't gotten any leads on it. #22094

@bsplosion
Copy link

One other fix, that works for me, is the following and after that everything works fine under windows.

Adding NX_NATIVE_COMMAND_RUNNER=false to the .env file at workspace root. The .env file was introduced with NX18.

Thanks so much for mentioning this. I was struggling to get NX to recognize the user environment copy of that variable but this solved it cleanly.

I assume once #22294 is merged and released we can remove that env variable again? Surprised this one wasn't caught during regression tests, seems like a very common use case.

Copy link

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 15, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.