Bind mount performance really slow using JEST testing framework

[MaximeHeckel]

Description

I'm trying to leverage docker to be a the core of some frontend development flow. To do that I'm trying to run some code from a local folder inside a node:7.4-slim container by mounting the folder in a volume. In general file watching and serving files works as expected and doesn't have any performance hit from the volume mounting, but running unit tests with the JEST framework (https://facebook.github.io/jest/) is extremely slow.

An issue has been opened a few months ago on the JEST repository but I think opening one here might give the core team a better idea of the problem we're facing.
Link of the issue

Steps to reproduce the issue:

1. Considering you have npm installed on your machine run npm install -g create-react-app
2. create-react-app test && cd test
3. docker run --rm -v $(pwd):/app -w /app node:7.4-slim bash -c "npm install && npm run test"
4. To compare the performance with the previous result just run locally npm run test

Describe the results you received:

 PASS  src/App.test.js
  ✓ renders without crashing (33ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        7.334s 
Ran all test suites.

Watch Usage
 › Press p to filter by a filename regex pattern.
 › Press q to quit watch mode.
 › Press Enter to trigger a test run.

To run one simple unit tests it takes more than 7s

Describe the results you expected:
In local the performance are really fast here's the output of the test task:

 PASS  src/App.test.js
  ✓ renders without crashing (26ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.229s
Ran all test suites.

Watch Usage
 › Press p to filter by a filename regex pattern.
 › Press q to quit watch mode.
 › Press Enter to trigger a test run.

In local it takes only 1s

Output of docker version:

❯ docker version
Client:
 Version:      17.03.0-ce-rc1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   ce07fb6
 Built:        Mon Feb 20 10:12:38 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.03.0-ce-rc1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   ce07fb6
 Built:        Mon Feb 20 10:12:38 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

❯ docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 352
Server Version: 17.03.0-ce-rc1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: hee5jcxna09x87as6jjtiv6ks
 Is Manager: true
 ClusterID: 8prdubyum2e07vr0vklll66dq
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  External CAs:
    cfssl: https://172.17.0.1:12381/api/v1/cfssl/sign
 Node Address: 172.17.0.1
 Manager Addresses:
  172.17.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.11-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.787 GiB
Name: moby
ID: 2KOB:WR4E:XVOY:OT3G:K4X4:2WMZ:6QWX:LBHQ:KF6Z:SZIR:ZBT2:IKNZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 33
 Goroutines: 131
 System Time: 2017-02-28T21:04:59.566559501Z
 EventsListeners: 1
Username: maximeheckel
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

I'm running the previously described steps on my MacBook Pro 2015 13 inch i7 16GB of memory on Docker for Mac

[barat]

It's known issue ...
#77
I even wrote an article about this:
http://espeo.eu/blog/docker-for-mac-performance-tweaks/

Hopefully docker-sync is using volumens now and You may use it as -v source argument

[dsheets]

@MaximeHeckel Thanks for the easily reproducible performance test case. Looking at some basic access statistics, it appears that whatever is executed from npm run test does about 300,000 file system operations resulting in more than 800,000 vsock messages before printing the Watch Usage message. This may be due to traversing all of node_modules (multiple times?) to set up inotify watches on subdirectories. I measure ~132s from docker run invocation until the Watch Usage message but the test system reports 6.2s elapsed so there may be some overhead here. Is it possible to only run the test without doing the rest? How do you normally use this software? Just npm run test and let it watch?

I also see that the access is highly parallel (up to 6 file system operations outstanding simultaneously). Perhaps surprisingly, this workload performs better with 1 vcpu allocated to Docker for Mac compared to 4 with test system-reported time of 5s (vs 6.2s) and end-to-end time of 98s (vs 132s). We've been investigating this issue for several months (in parallel to other work) and the cause is a combination of the Linux scheduler, thread migration policy, and HyperKit interprocessor interrupt overhead. We have some modifications to Linux configuration and our in-VM thread use which may alleviate this multiprocessing bottleneck but those changes require more validation before being deployed which is currently prioritized below other performance improvement work. What version of macOS are you running?

The file system team has some improvements to performance nearing release which may help with this use case so I'm going to leave this issue open to track improvement of your usage. Thanks for your report!

[MaximeHeckel]

@dsheets Yes there's a way to run the test without the watch mode

• export CI=true
• npm run test --coverage
  But in general our use case is to let it watch for changes.

I ran it in non watch mode in docker and here are the results:

❯ docker run --rm -v $(pwd):/app -w /app -e CI=true node:7.4-slim bash -c "npm run test --coverage"
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~pretest: [email protected]
npm info lifecycle [email protected]~test: [email protected]

> [email protected] test /app
> node scripts/test.js --env=jsdom

 PASS  src/App.test.js
  ✓ renders without crashing (23ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        7.805s
Ran all test suites.
npm info lifecycle [email protected]~posttest: [email protected]
npm info ok

Maybe you'll get more info from this.

I'm running macOS Sierra 10.12.3, but I had this same issue on 10.11 as well.

[tlrjs]

I'm having the same issue. Running inside of docker for mac my jest tests take 72.95 seconds and outside of docker the tests take 18.754 seconds.

[sfwn]

In my opinion, maybe the storage strategy docker used causes the problem.

When i run mvn clean package inside official maven container cost 3 times long compare to just running this cmd in my host.

reference: https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/#the-copy-on-write-strategy

[docker-robott]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[LondonAppDev]

In-case it helps anyone, I managed to improve the performance situation when using docker-compose by adding an extra volume for node_modules:

    volumes:
      - .:/code
      - /code/node_modules

After doing this the speed went from 17 seconds (for 3 tests) to about 3 seconds.

I am running macOS 10.13.4 and using Docker 18.03.1-ce.

[clmsnskr]

@LondonAppDev Yes! This! Don't need the host machine's node_modules folder in the container anyway ( especially if the host machine is a different kernel than the container)

[YRM64]

Thanks for sharing @MaximeHeckel. Can you please share with the development community the modifications to Linux configurations and any possible changes to the configuration for VMs as soon as they become available. We know how the validation process works with some performance issues taking priority over others. Please hit us back at this posting as-soon-as the information becomes available. Thanks in advance. I have a MacOs, version 12.6.3. I would like to have that information specific to my operating system.

[cdomigan]

@clmsnskr without node_modules mounted, how are you avoiding errors from missing modules in your tests?

[LondonAppDev]

@cdomigan in my setup, the node modules are installed when the docker image is built, and to install them I use docker-compose run app yarn add <dependency>.

[vschoener]

@LondonAppDev

I just tried it and it says : sh: jest: not found

I also use docker-compose and alpine node image.

Maybe I should install dependencies during the build process.

[fabioelizandro]

It's something related to jest multi thread. Change your docker to use only one CPU at time and then run the tests again. My tests went from 14s to 3s

[Vinchens00-zz]

@fabioelizandro is correct.

There is a way to keep global docker configuration as it's.
We just added --maxWorkers=1 flag to jest and it solved the issue.

[travisluong]

The only thing that has sped up the tests for me was using a named volume on the node_modules folder.

    volumes:
      - ./server:/usr/src/app/
      - server_node_modules:/usr/src/app/node_modules

volumes:
  server_node_modules:

[matt-continuousdelta]

When I use an anonymous or named volume as described at #1358 (comment) or #1358 (comment) the resulting node_modules directory inside the container is owned by root instead of the node user and isn't writable by the node user. What am I missing?

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked