Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System Process High CPU use #12939

Closed
hernandp opened this issue Sep 1, 2022 · 6 comments
Closed

System Process High CPU use #12939

hernandp opened this issue Sep 1, 2022 · 6 comments

Comments

@hernandp
Copy link

hernandp commented Sep 1, 2022

  • [X ] I have tried with the latest version of Docker Desktop
  • [ X] I have tried disabling enabled experimental features
  • [ X] I have uploaded Diagnostics
  • Diagnostics ID: 682F92B3-3118-417C-806D-B355ED8FDCD8/20220901031429

Actual behavior

System Process CPU use >15%. Using ProcExp shows a thread of NTOSKRNL!Ordinal25+0x1730 address consuming cycles continuosly.
No containers running.
Docker does not exit immediately if I request by UI.

Expected behavior

I expect low CPU use without containers running.

Information

  • Windows Version: 21H2 (19044.1889)
  • Docker Desktop Version: 4.11.1
  • WSL2 or Hyper-V backend? WSL2
  • Are you running inside a virtualized Windows e.g. on a cloud server or a VM: No

Output of & "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" check

[2022-09-01T03:24:50.490070700Z][com.docker.diagnose.exe][I] set path configuration to OnHost
Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0031: does the Docker API work?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0001: is the application running?
[SKIP] DD0018: does the host support virtualization?
[PASS] DD0002: does the bootloader have virtualization enabled?
[PASS] DD0017: can a VM be started?
[PASS] DD0024: is WSL installed?
[PASS] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0005: is the user in the docker-users group?
[PASS] DD0007: is the backend responding?
[FAIL] DD0014: are the backend processes running? 1 error occurred:
        * vpnkit.exe is not running


[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[PASS] DD0006: is the Docker Desktop Service responding?
[FAIL] DD0012: is the VM networking working? network checks failed: failed to ping host: exit status 1
[2022-09-01T03:25:05.314677400Z][com.docker.diagnose.exe][I] ipc.NewClient: c3802608-diagnose-network -> \\.\pipe\dockerDiagnosticd diagnosticsd
[common/pkg/diagkit/gather/diagnose.runIsVMNetworkingOK()
[       common/pkg/diagkit/gather/diagnose/network.go:34 +0xd9
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0xdecfe0)
[       common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0xdecfe0)
[       common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x2?, 0xdecfe0)
[       common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0xdecfe0, 0xc000479728)
[       common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0xcb?, 0xc000479728)
[       common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x7a3a40?, 0xc0002df890)
[       common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0xded5e0, 0x82b4fed00000010?, {0xc0002dfb20, 0x1, 0x1})
[       common/pkg/diagkit/gather/diagnose/run.go:16 +0x1ca
[main.checkCmd({0xc00007a3d0?, 0xc00007a3d0?, 0x4?}, {0x0, 0x0})
[       common/cmd/com.docker.diagnose/main.go:133 +0x105
[main.main()
[       common/cmd/com.docker.diagnose/main.go:99 +0x288
[2022-09-01T03:25:05.315334700Z][com.docker.diagnose.exe][I] (1cce39e7) c3802608-diagnose-network C->S diagnosticsd POST /check-network-connectivity: {"ips":["172.25.160.1","192.168.56.1","192.168.0.30","172.19.80.1","172.29.176.1","172.17.224.1","172.22.224.1"]}
[2022-09-01T03:25:05.883559100Z][com.docker.diagnose.exe][W] (1cce39e7) c3802608-diagnose-network C<-S dca7b135-diagnosticsd POST /check-network-connectivity (568.272ms): failed to ping host: exit status 1

[FAIL] DD0032: do Docker networks overlap with host IPs? network bridge has subnet 172.17.0.0/16 which overlaps with host IP 172.17.224.1
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?

Please investigate the following 3 issues:

1 : The test: are the backend processes running?
    Failed with: 1 error occurred:
        * vpnkit.exe is not running



Not all of the backend processes are running.

2 : The test: is the VM networking working?
    Failed with: network checks failed: failed to ping host: exit status 1

VM seems to have a network connectivity issue. Please check your host firewall and anti-virus settings in case they are blocking the VM.

3 : The test: do Docker networks overlap with host IPs?
    Failed with: network bridge has subnet 172.17.0.0/16 which overlaps with host IP 172.17.224.1

If the subnet used by a Docker network overlaps with an IP used by the host, then containers
won't be able to contact the overlapping IP addresses.

Please try configuring the IP address range used by networks: in your docker-compose.yml.
See https://docs.docker.com/compose/compose-file/compose-file-v2/#ipv4_address-ipv6_address

Steps to reproduce the behavior

Run Docker/WSL2 session e.g Terminal and wait until System process starts eating CPU.

@hernandp
Copy link
Author

hernandp commented Sep 9, 2022

The kernel thread 108 is using the CPU by calling repeatedly KeBalanceSetManager, which can be caused by memory pressure. I analyzed a memory dump of my system and realized there were a lot (over 30000) of WSL.EXE processes seemingly dead -but each with 80kb allocated mem.

They were all created by docker backend process, e.g:

0: kd> !process 688
Searching for Process with Cid == 688
PROCESS ffffe784296c5080
SessionId: 1 Cid: 0688 Peb: 4683bf000 ParentCid: 4498
DirBase: 4fa68e000 ObjectTable: 00000000 HandleCount: 0.
Image: wsl.exe
VadRoot 0000000000000000 Vads 0 Clone 0 Private 15. Modified 5. Locked 0.
DeviceMap ffffa90f59d5b1f0
Token ffffa90f72351060
ElapsedTime 20:36:27.640
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (16, 50, 345) (64KB, 200KB, 1380KB)
PeakWorkingSetSize 1902
VirtualSize 0 Mb
PeakVirtualSize 2101308 Mb
PageFaultCount 2021
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 19
Job ffffe784332c20a0

Parent of that process?

0: kd> !process 4498

Searching for Process with Cid == 4498
PROCESS ffffe7842c84b300
SessionId: 1 Cid: 4498 Peb: ad5622d000 ParentCid: 2d74
DirBase: 513da3000 ObjectTable: ffffa90f63765800 HandleCount: 41579.
Image: com.docker.backend.exe
VadRoot ffffe7842c6bbd20 Vads 179 Clone 0 Private 6125. Modified 85188. Locked 5.
DeviceMap ffffa90f59d5b1f0
Token ffffa90f633525f0
ElapsedTime 20:51:33.928
UserTime 00:00:49.046
KernelTime 00:02:00.531
QuotaPoolUsage[PagedPool] 929952
QuotaPoolUsage[NonPagedPool] 35440
Working Set Sizes (now,min,max) (10980, 50, 345) (43920KB, 200KB, 1380KB)
PeakWorkingSetSize 14367
VirtualSize 4947 Mb
PeakVirtualSize 4959 Mb
PageFaultCount 872947
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 10038
Job ffffe784332c20a0

Look at handle count :

image

Maybe this is caused by some problem where docker backend creates/kills a lot of dummy WSL.EXE processes in rapid succession.

@hernandp
Copy link
Author

hernandp commented Sep 9, 2022

Related issue #12916

@nicks
Copy link

nicks commented Sep 9, 2022

does the problem persist if you upgrade to Docker Desktop v4.12? This sounds a lot like - #12877

@hernandp
Copy link
Author

Upgraded. Several hours without any issue. Let's hope 4.12 fixes it , thank you. I'll look again the next week as I'll be using Docker+WSL a lot.

@nicks
Copy link

nicks commented Sep 26, 2022

Hooray! Closing as resolved.

@nicks nicks closed this as completed Sep 26, 2022
@docker-robott
Copy link
Collaborator

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

@docker docker locked and limited conversation to collaborators Oct 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants