Memory issues

[pedroxns]

The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Also, before reporting a new issue, please make sure that:

• You read carefully the documentation and frequently asked questions.
• You searched for a similar issue and this is not a duplicate of an existing one.
• This issue is not related to NGC, otherwise, please use the devtalk forums instead.
• You went through the troubleshooting steps.

---

1. Issue or feature description

I'm running a Debian VM on proxmox with gpu passthrough;
Everytime I start a container i receive an "out of memory" error message. I was trying it with a Quadro P400 and now changed to a RTX2060 but that eror persists.

2. Steps to reproduce the issue

Start a deepstack or a compreface container

3. Information to attach (optional if deemed irrelevant)

• Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info

I0117 15:18:51.313599 1426 nvc.c:376] initializing library context (version=1.11.0, build=c8f267be0bac1c654d59ad4ea5df907141149977)
I0117 15:18:51.313634 1426 nvc.c:350] using root /
I0117 15:18:51.313641 1426 nvc.c:351] using ldcache /etc/ld.so.cache
I0117 15:18:51.313646 1426 nvc.c:352] using unprivileged user 1000:1000
I0117 15:18:51.313662 1426 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0117 15:18:51.313721 1426 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
W0117 15:18:51.314926 1427 nvc.c:273] failed to set inheritable capabilities
W0117 15:18:51.314958 1427 nvc.c:274] skipping kernel modules load due to failure
I0117 15:18:51.315149 1428 rpc.c:71] starting driver rpc service
I0117 15:18:51.318126 1429 rpc.c:71] starting nvcgo rpc service
I0117 15:18:51.318885 1426 nvc_info.c:766] requesting driver information with ''
I0117 15:18:51.320030 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.525.78.01
I0117 15:18:51.320122 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvoptix.so.525.78.01
I0117 15:18:51.320164 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.525.78.01
I0117 15:18:51.320190 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.525.78.01
I0117 15:18:51.320216 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.525.78.01
I0117 15:18:51.320250 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.525.78.01
I0117 15:18:51.320282 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.525.78.01
I0117 15:18:51.320308 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.525.78.01
I0117 15:18:51.320334 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.525.78.01
I0117 15:18:51.320372 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.525.78.01
I0117 15:18:51.320400 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.525.78.01
I0117 15:18:51.320430 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.525.78.01
I0117 15:18:51.320457 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.525.78.01
I0117 15:18:51.320480 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.525.78.01
I0117 15:18:51.320503 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.525.78.01
I0117 15:18:51.320521 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.525.78.01
I0117 15:18:51.320537 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.525.78.01
I0117 15:18:51.320562 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.525.78.01
I0117 15:18:51.320605 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.525.78.01
I0117 15:18:51.320720 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libcudadebugger.so.525.78.01
I0117 15:18:51.320770 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.525.78.01
I0117 15:18:51.320831 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.525.78.01
I0117 15:18:51.320864 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.525.78.01
I0117 15:18:51.320893 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.525.78.01
I0117 15:18:51.320922 1426 nvc_info.c:173] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.525.78.01
W0117 15:18:51.320940 1426 nvc_info.c:399] missing library libnvidia-nscq.so
W0117 15:18:51.320947 1426 nvc_info.c:399] missing library libnvidia-fatbinaryloader.so
W0117 15:18:51.320951 1426 nvc_info.c:399] missing library libnvidia-pkcs11.so
W0117 15:18:51.320955 1426 nvc_info.c:399] missing library libnvidia-ifr.so
W0117 15:18:51.320958 1426 nvc_info.c:399] missing library libnvidia-cbl.so
W0117 15:18:51.320963 1426 nvc_info.c:403] missing compat32 library libnvidia-ml.so
W0117 15:18:51.320966 1426 nvc_info.c:403] missing compat32 library libnvidia-cfg.so
W0117 15:18:51.320974 1426 nvc_info.c:403] missing compat32 library libnvidia-nscq.so
W0117 15:18:51.320977 1426 nvc_info.c:403] missing compat32 library libcuda.so
W0117 15:18:51.320980 1426 nvc_info.c:403] missing compat32 library libcudadebugger.so
W0117 15:18:51.320984 1426 nvc_info.c:403] missing compat32 library libnvidia-opencl.so
W0117 15:18:51.320989 1426 nvc_info.c:403] missing compat32 library libnvidia-ptxjitcompiler.so
W0117 15:18:51.320992 1426 nvc_info.c:403] missing compat32 library libnvidia-fatbinaryloader.so
W0117 15:18:51.320995 1426 nvc_info.c:403] missing compat32 library libnvidia-allocator.so
W0117 15:18:51.320999 1426 nvc_info.c:403] missing compat32 library libnvidia-compiler.so
W0117 15:18:51.321003 1426 nvc_info.c:403] missing compat32 library libnvidia-pkcs11.so
W0117 15:18:51.321010 1426 nvc_info.c:403] missing compat32 library libnvidia-ngx.so
W0117 15:18:51.321014 1426 nvc_info.c:403] missing compat32 library libvdpau_nvidia.so
W0117 15:18:51.321018 1426 nvc_info.c:403] missing compat32 library libnvidia-encode.so
W0117 15:18:51.321022 1426 nvc_info.c:403] missing compat32 library libnvidia-opticalflow.so
W0117 15:18:51.321027 1426 nvc_info.c:403] missing compat32 library libnvcuvid.so
W0117 15:18:51.321033 1426 nvc_info.c:403] missing compat32 library libnvidia-eglcore.so
W0117 15:18:51.321037 1426 nvc_info.c:403] missing compat32 library libnvidia-glcore.so
W0117 15:18:51.321044 1426 nvc_info.c:403] missing compat32 library libnvidia-tls.so
W0117 15:18:51.321046 1426 nvc_info.c:403] missing compat32 library libnvidia-glsi.so
W0117 15:18:51.321055 1426 nvc_info.c:403] missing compat32 library libnvidia-fbc.so
W0117 15:18:51.321058 1426 nvc_info.c:403] missing compat32 library libnvidia-ifr.so
W0117 15:18:51.321062 1426 nvc_info.c:403] missing compat32 library libnvidia-rtcore.so
W0117 15:18:51.321067 1426 nvc_info.c:403] missing compat32 library libnvoptix.so
W0117 15:18:51.321068 1426 nvc_info.c:403] missing compat32 library libGLX_nvidia.so
W0117 15:18:51.321077 1426 nvc_info.c:403] missing compat32 library libEGL_nvidia.so
W0117 15:18:51.321083 1426 nvc_info.c:403] missing compat32 library libGLESv2_nvidia.so
W0117 15:18:51.321089 1426 nvc_info.c:403] missing compat32 library libGLESv1_CM_nvidia.so
W0117 15:18:51.321093 1426 nvc_info.c:403] missing compat32 library libnvidia-glvkspirv.so
W0117 15:18:51.321098 1426 nvc_info.c:403] missing compat32 library libnvidia-cbl.so
I0117 15:18:51.321234 1426 nvc_info.c:299] selecting /usr/bin/nvidia-smi
I0117 15:18:51.321254 1426 nvc_info.c:299] selecting /usr/bin/nvidia-debugdump
I0117 15:18:51.321273 1426 nvc_info.c:299] selecting /usr/bin/nvidia-persistenced
I0117 15:18:51.321297 1426 nvc_info.c:299] selecting /usr/bin/nvidia-cuda-mps-control
I0117 15:18:51.321315 1426 nvc_info.c:299] selecting /usr/bin/nvidia-cuda-mps-server
W0117 15:18:51.324656 1426 nvc_info.c:425] missing binary nv-fabricmanager
W0117 15:18:51.324749 1426 nvc_info.c:349] missing firmware path /lib/firmware/nvidia/525.78.01/gsp.bin
I0117 15:18:51.324801 1426 nvc_info.c:529] listing device /dev/nvidiactl
I0117 15:18:51.324807 1426 nvc_info.c:529] listing device /dev/nvidia-uvm
I0117 15:18:51.324810 1426 nvc_info.c:529] listing device /dev/nvidia-uvm-tools
I0117 15:18:51.324813 1426 nvc_info.c:529] listing device /dev/nvidia-modeset
W0117 15:18:51.324841 1426 nvc_info.c:349] missing ipc path /var/run/nvidia-persistenced/socket
W0117 15:18:51.324862 1426 nvc_info.c:349] missing ipc path /var/run/nvidia-fabricmanager/socket
W0117 15:18:51.324879 1426 nvc_info.c:349] missing ipc path /tmp/nvidia-mps
I0117 15:18:51.324886 1426 nvc_info.c:822] requesting device information with ''
I0117 15:18:51.331250 1426 nvc_info.c:713] listing device /dev/nvidia0 (GPU-d0b2b12b-a223-ad5a-6661-e65ffde8f84b at 00000000:00:10.0)
NVRM version:   525.78.01
CUDA version:   12.0

Device Index:   0
Device Minor:   0
Model:          NVIDIA GeForce RTX 2060
Brand:          GeForce
GPU UUID:       GPU-d0b2b12b-a223-ad5a-6661-e65ffde8f84b
Bus Location:   00000000:00:10.0
Architecture:   7.5
I0117 15:18:51.331286 1426 nvc.c:434] shutting down library context
I0117 15:18:51.331341 1429 rpc.c:95] terminating nvcgo rpc service
I0117 15:18:51.331804 1426 rpc.c:135] nvcgo rpc service terminated successfully
I0117 15:18:51.333713 1428 rpc.c:95] terminating driver rpc service
I0117 15:18:51.333969 1426 rpc.c:135] driver rpc service terminated successfully

• Kernel version from uname -a

Linux debian 5.10.0-20-amd64 NVIDIA/nvidia-docker#1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux``` 

 - [ ] Any relevant kernel output lines from `dmesg`
 - [ ] Driver information from `nvidia-smi -a`
 ```Timestamp                                 : Tue Jan 17 12:21:08 2023
Driver Version                            : 525.78.01
CUDA Version                              : 12.0

Attached GPUs                             : 1
GPU 00000000:00:10.0
    Product Name                          : NVIDIA GeForce RTX 2060
    Product Brand                         : GeForce
    Product Architecture                  : Turing
    Display Mode                          : Disabled
    Display Active                        : Disabled
    Persistence Mode                      : Disabled
    MIG Mode
        Current                           : N/A
        Pending                           : N/A
    Accounting Mode                       : Disabled
    Accounting Mode Buffer Size           : 4000
    Driver Model
        Current                           : N/A
        Pending                           : N/A
    Serial Number                         : N/A
    GPU UUID                              : GPU-d0b2b12b-a223-ad5a-6661-e65ffde8f84b
    Minor Number                          : 0
    VBIOS Version                         : 90.04.63.40.55
    MultiGPU Board                        : No
    Board ID                              : 0x10
    Board Part Number                     : N/A
    GPU Part Number                       : 1E89-150-A1
    Module ID                             : 1
    Inforom Version
        Image Version                     : G001.0000.02.04
        OEM Object                        : 1.1
        ECC Object                        : N/A
        Power Management Object           : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GSP Firmware Version                  : N/A
    GPU Virtualization Mode
        Virtualization Mode               : Pass-Through
        Host VGPU Mode                    : N/A
    IBMNPU
        Relaxed Ordering Mode             : N/A
    PCI
        Bus                               : 0x00
        Device                            : 0x10
        Domain                            : 0x0000
        Device Id                         : 0x1E8910DE
        Bus Id                            : 00000000:00:10.0
        Sub System Id                     : 0x20683842
        GPU Link Info
            PCIe Generation
                Max                       : 3
                Current                   : 1
                Device Current            : 1
                Device Max                : 3
                Host Max                  : N/A
            Link Width
                Max                       : 16x
                Current                   : 16x
        Bridge Chip
            Type                          : N/A
            Firmware                      : N/A
        Replays Since Reset               : 0
        Replay Number Rollovers           : 0
        Tx Throughput                     : 0 KB/s
        Rx Throughput                     : 0 KB/s
        Atomic Caps Inbound               : N/A
        Atomic Caps Outbound              : N/A
    Fan Speed                             : 0 %
    Performance State                     : P8
    Clocks Throttle Reasons
        Idle                              : Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    FB Memory Usage
        Total                             : 6144 MiB
        Reserved                          : 217 MiB
        Used                              : 0 MiB
        Free                              : 5926 MiB
    BAR1 Memory Usage
        Total                             : 256 MiB
        Used                              : 2 MiB
        Free                              : 254 MiB
    Compute Mode                          : Default
    Utilization
        Gpu                               : 0 %
        Memory                            : 0 %
        Encoder                           : 0 %
        Decoder                           : 0 %
    Encoder Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    FBC Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    Ecc Mode
        Current                           : N/A
        Pending                           : N/A
    ECC Errors
        Volatile
            SRAM Correctable              : N/A
            SRAM Uncorrectable            : N/A
            DRAM Correctable              : N/A
            DRAM Uncorrectable            : N/A
        Aggregate
            SRAM Correctable              : N/A
            SRAM Uncorrectable            : N/A
            DRAM Correctable              : N/A
            DRAM Uncorrectable            : N/A
    Retired Pages
        Single Bit ECC                    : N/A
        Double Bit ECC                    : N/A
        Pending Page Blacklist            : N/A
    Remapped Rows                         : N/A
    Temperature
        GPU Current Temp                  : 37 C
        GPU Shutdown Temp                 : 93 C
        GPU Slowdown Temp                 : 90 C
        GPU Max Operating Temp            : 88 C
        GPU Target Temperature            : 83 C
        Memory Current Temp               : N/A
        Memory Max Operating Temp         : N/A
    Power Readings
        Power Management                  : Supported
        Power Draw                        : 9.02 W
        Power Limit                       : 170.00 W
        Default Power Limit               : 170.00 W
        Enforced Power Limit              : 170.00 W
        Min Power Limit                   : 125.00 W
        Max Power Limit                   : 170.00 W
    Clocks
        Graphics                          : 300 MHz
        SM                                : 300 MHz
        Memory                            : 405 MHz
        Video                             : 540 MHz
    Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Default Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Deferred Clocks
        Memory                            : N/A
    Max Clocks
        Graphics                          : 2145 MHz
        SM                                : 2145 MHz
        Memory                            : 7001 MHz
        Video                             : 1950 MHz
    Max Customer Boost Clocks
        Graphics                          : N/A
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A
    Voltage
        Graphics                          : N/A
    Fabric
        State                             : N/A
        Status                            : N/A
    Processes                             : None

• Docker version from docker version

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:22 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.15
  GitCommit:        5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Nome                          Versão       Arquitectura Descrição
+++-=============================-============-============-=====================================================
ii  libnvidia-container-tools     1.11.0-1     amd64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64    1.11.0-1     amd64        NVIDIA container runtime library
ii  nvidia-container-runtime      3.11.0-1     all          NVIDIA container runtime
un  nvidia-container-runtime-hook <nenhuma>    <nenhuma>    (nenhuma descrição disponível)
ii  nvidia-container-toolkit      1.11.0-1     amd64        NVIDIA Container toolkit
ii  nvidia-container-toolkit-base 1.11.0-1     amd64        NVIDIA Container Toolkit Base
un  nvidia-docker                 <nenhuma>    <nenhuma>    (nenhuma descrição disponível)
ii  nvidia-docker2                2.11.0-1     all          nvidia-docker CLI wrapper```
- [ ] NVIDIA container library version from `nvidia-container-cli -V`
```cli-version: 1.11.0
lib-version: 1.11.0
build date: 2022-09-06T09:21+00:00
build revision: c8f267be0bac1c654d59ad4ea5df907141149977
build compiler: x86_64-linux-gnu-gcc-8 8.3.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections```
- [ ] NVIDIA container library logs (see [troubleshooting](https://github.com/NVIDIA/nvidia-docker/wiki/Troubleshooting))
```{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:38:16-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:38:26-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:38:26-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:38:26-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:38:32-03:00"}
{"level":"info","msg":"Using OCI specification file path: /run/containerd/io.containerd.runtime.v2.task/moby/d564955d89f09cdacbb0d1e731acc41f2e859d15ff83a6d52257b7ef58212b86/config.json",>{"level":"info","msg":"Auto-detected mode as 'legacy'","time":"2023-01-17T12:38:32-03:00"}
{"level":"info","msg":"Using prestart hook path: /usr/bin/nvidia-container-runtime-hook","time":"2023-01-17T12:38:32-03:00"}
{"level":"info","msg":"Applied required modification to OCI specification","time":"2023-01-17T12:38:32-03:00"}
{"level":"info","msg":"Forwarding command to runtime","time":"2023-01-17T12:38:32-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:38:33-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:38:59-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:39:09-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:39:09-03:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-01-17T12:39:09-03:00"}

• Docker command, image and tag used
  Docker compose

version: '3.7'
services:
  deepstack:
    container_name: deepstack
    restart: unless-stopped
    image: deepquestai/deepstack:gpu
    runtime: nvidia
    ports:
      - '5005:5000'
    environment:
      - VISION-FACE=True
    #  - VISION-DETECTION=True
     # - MODE=High
      #API-KEY= ''
    volumes:
      - /home/deepstack/models:/modelstore/detection

--

GNU nano 5.4                                                                                         
version: '3.7'

volumes:
  double-take:
  deepstack:

services:
  double-take:
    container_name: double-take
    image: jakowenko/double-take:latest
    restart: unless-stopped
    volumes:
      - /home/dtake/double-take:/.storage
    ports:
      - '3000:3000'

  deepstack:
    container_name: deepstack_dtake
    restart: unless-stopped
    image: deepquestai/deepstack:gpu
    runtime: nvidia
    ports:
      - '5000:5000'
    environment:
      - VISION-FACE=True
     # - VISION-DETECTION=True
      - MODE=Low
      #API-KEY= ''
    volumes:
      - /home/dtake/deepstack/models:/modelstore/detection