Unable to start container on Linux 6.7

[quinnjr]

Currently unable to start the container on Arch Linux as the host OS. The dump files for the failing sqlservr process don't really provide any insight as to why:

docker compose logs db
db-1  | SQL Server 2022 will run as non-root by default.
db-1  | This container is running as user mssql.
db-1  | Your master database file is owned by mssql.
db-1  | To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
db-1  | This program has encountered a fatal error and cannot continue running at Mon Jan 15 18:19:00 2024
db-1  | The following diagnostic information is available:
db-1  | 
db-1  |          Reason: 0x00000001
db-1  |          Signal: SIGABRT - Aborted (6)
db-1  |           Stack:
db-1  |                  IP               Function
db-1  |                  ---------------- --------------------------------------
db-1  |                  000064eb280a3ce1 std::__1::bad_function_call::~bad_function_call()+0x96661
db-1  |                  000064eb280a36a6 std::__1::bad_function_call::~bad_function_call()+0x96026
db-1  |                  000064eb280a2c2f std::__1::bad_function_call::~bad_function_call()+0x955af
db-1  |                  00007c18f8810520 __sigaction+0x50
db-1  |                  00007c18f88649fc pthread_kill+0x12c
db-1  |                  00007c18f8810476 raise+0x16
db-1  |                  00007c18f87f67f3 abort+0xd3
db-1  |                  000064eb28074d96 std::__1::bad_function_call::~bad_function_call()+0x67716
db-1  |                  000064eb280b15b4 std::__1::bad_function_call::~bad_function_call()+0xa3f34
db-1  |                  000064eb280df318 std::__1::bad_function_call::~bad_function_call()+0xd1c98
db-1  |                  000064eb280df0fa std::__1::bad_function_call::~bad_function_call()+0xd1a7a
db-1  |                  000064eb2807b20a std::__1::bad_function_call::~bad_function_call()+0x6db8a
db-1  |                  000064eb2807ae80 std::__1::bad_function_call::~bad_function_call()+0x6d800
db-1  |         Process: 10 - sqlservr
db-1  |          Thread: 157 (application thread 0x264)
db-1  |     Instance Id: 83ef72ce-1100-44c4-913c-45d0df61ae44
db-1  |        Crash Id: 05e56c63-9bd1-47db-b3d5-c1f58cebd578
db-1  |     Build stamp: a9299dd605c652a3cea4246273441bcfaf48afb4b482ab9dc43771eecaf6600b
db-1  |    Distribution: Ubuntu 22.04.3 LTS
db-1  |      Processors: 32
db-1  |    Total Memory: 67119079424 bytes
db-1  |       Timestamp: Mon Jan 15 18:19:00 2024
db-1  |      Last errno: 2
db-1  | Last errno text: No such file or directory
db-1  | Capturing a dump of 10
db-1  | Successfully captured dump: /var/opt/mssql/log/core.sqlservr.1_15_2024_18_19_0.10
db-1  | Executing: /opt/mssql/bin/handle-crash.sh with parameters
db-1  |      handle-crash.sh
db-1  |      /opt/mssql/bin/sqlservr
db-1  |      10
db-1  |      /opt/mssql/bin
db-1  |      /var/opt/mssql/log/
db-1  |      
db-1  |      83ef72ce-1100-44c4-913c-45d0df61ae44
db-1  |      05e56c63-9bd1-47db-b3d5-c1f58cebd578
db-1  |      
db-1  |      /var/opt/mssql/log/core.sqlservr.1_15_2024_18_19_0.10
db-1  | 
db-1  | Ubuntu 22.04.3 LTS
db-1  | Capturing core dump and information to /var/opt/mssql/log...

Docker-compose file:

version: '3'
services:
  db:
    image: 'mcr.microsoft.com/mssql/server:2022-latest'
    environment:
      - ACCEPT_EULA=Y
      - MSSQL_SA_PASSWORD=<there would be a password here>
      - MSSQL_PID=Developer
    volumes:
      - ./logs:/var/opt/mssql/log
      - ./data:/var/opt/mssql/data
    ports:
      - 1433:1433

Docker logs and data directory are set as UID:GID 10001:10001.

[erikbozic]

I have the same issue. Found that it's the 6.7 kernel update. (#858 (comment))

Rolling back to 6.6.10 makes it work again.

[thomasvm]

I experienced the same behavior today. First my existing container grew in size very quickly. I tried creating other containers but they all failed with the above message.

It took me a while to figure out that downgrading my kernel fixes the issue, but downgrading to 6.6.11 did the trick.

[unlogicalcode]

I can also confirm, I have the same behaviour.
It works with Kernel 6.6 and with 6.7 I get a similiar Message as above.

[quinnjr]

I downgraded my kernel and the container now functions.

Is this limited to just this container or docker needing to update something to be compatible with the 6.7 kernel?

[huestack]

I have same problem running container in Podman, but the Docker container is running without any problem. I simply pulled the image sudo podman pull mcr.microsoft.com/mssql/server:2022-latest, and ran it:

sudo podman run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=Str0ngPass!" -p 1433:1433 --name sql-test --hostname sql-test -d  mcr.microsoft.com/mssql/server:2022-latest

Attached is a log file.
sql-test.log

[LJFloor]

Can confirm on Arch Linux, both the docker images for versions 2017, 2019 and 2022 and the AUR version give the same result.

Last errno text: No such file or directory

After downgrading the kernel to version 6.6.10-arch1-1 it starts successfully.

[CodeKJ]

I can confirm this on Nobara 39 with 6.7.0 kernel. Exactly same issue for 2017, 2019, 2022 mssql.
6.6.9 works fine.

[erikbozic]

It seems like this was solved in the aur repo package mssql-server: https://aur.archlinux.org/packages/mssql-server#comment-953063.
However I'm still having trouble building the needed dependency to verify...

[kshpytsya]

For what it is worth:

running Gentoo with custom 6.7.x kernel.
It looks like it fails trying to access cgroup v1 "/sys/fs/cgroup/memory/memory.limit_in_bytes".
I suspect that switching to cgroup to "hybrid" would fix the issue but I am not up to rebooting my machine now.

$ docker run -it --rm -e ACCEPT_EULA=Y -e MSSQL_PID=Developer mcr.microsoft.com/mssql/server:2022-latest -- /bin/bash
sleep 1000

in another terminal, run

ps fax|less
# find pid of bash which is parent of sleep
sudo strace -o mssql.strace -f -s1000 -p <bash-in-mssql-docker>

return to the first terminal, Ctrl-C the sleep and run /opt/mssql/bin/sqlservr. Run /opt/mssql/bin/sqlservr and wait for it to crash. Go to the seconf terminal, interrupt strace.

$ grep -P '"/(proc|sys).*ENOENT' mssql.strace
9999 openat(AT_FDCWD, "/sys/fs/cgroup/memory/memory.limit_in_bytes", O_RDONLY) = -1 ENOENT (No such file or directory)

[ibauersachs]

I think the ENOENT is not the issue, especially not /sys/fs/cgroup/memory/memory.limit_in_bytes since this doesn't exist on Kernel 6.6.13 either, and mssql runs fine there.
My crashlogs on 6.7.1 showed Invalid argument / 22 / EINVAL:

This program has encountered a fatal error and cannot continue running at Mon Jan 22 18:09:17 2024
The following diagnostic information is available:

         Reason: 0x00000001
         Signal: SIGABRT - Aborted (6)
          Stack:
                 IP               Function
                 ---------------- --------------------------------------
                 0000613cdff2ace1 std::__1::bad_function_call::~bad_function_call()+0x96661
                 0000613cdff2a6a6 std::__1::bad_function_call::~bad_function_call()+0x96026
                 0000613cdff29c2f std::__1::bad_function_call::~bad_function_call()+0x955af
                 0000753f7ee4d520 __sigaction+0x50
                 0000753f7eea19fc pthread_kill+0x12c
                 0000753f7ee4d476 raise+0x16
                 0000753f7ee337f3 abort+0xd3
                 0000613cdfefbd96 std::__1::bad_function_call::~bad_function_call()+0x67716
        Process: 10 - sqlservr
         Thread: 161 (application thread 0x278)
    Instance Id: ba778b4b-ea20-4f3c-98fa-2002d4c8e68c
       Crash Id: 3674de73-5de7-494e-8530-2520421dd97f
    Build stamp: a9299dd605c652a3cea4246273441bcfaf48afb4b482ab9dc43771eecaf6600b
   Distribution: Ubuntu 22.04.3 LTS
     Processors: 16
   Total Memory: 29180137472 bytes
      Timestamp: Mon Jan 22 18:09:17 2024
     Last errno: 22
Last errno text: Invalid argument

[CryptoSiD]

The problem is still there with kernel 6.7.2

[Green0wl]

same problem on 6.7.1-arch1-1

[GieltjE]

As a bad side effect the lsof process it spawns starts eating a core

[fbrosseau]

Hello,

The issue has been identified and should be fixed in the next SQL Server 2022 CU, but we cannot commit to a specific CU release or timeline as sometimes plans can unexpectedly change. No further investigation or data points should be needed for this, but thank you all for reporting and for looking for potential causes.

It is unrelated to cgroups, and at first glance it might be a kernel bug (but do not quote me on this) - it appears that as of 6.7, mmap without MAP_FIXED may sometimes ignore the address hint even if the hinted region is in fact available. I have not investigated the kernel side of things further, but I think it might be related to this series of changes and/or its preceding/following changes.

Knowing this, I cannot think of any workaround other than sticking to 6.6 in the meantime.

[vermarine]

Hello,

The issue has been identified and should be fixed in the next SQL Server 2022 CU, but we cannot commit to a specific CU release or timeline as sometimes plans can unexpectedly change. No further investigation or data points should be needed for this, but thank you all for reporting and for looking for potential causes.

It is unrelated to cgroups, and at first glance it might be a kernel bug (but do not quote me on this) - it appears that as of 6.7, mmap without MAP_FIXED may sometimes ignore the address hint even if the hinted region is in fact available. I have not investigated the kernel side of things further, but I think it might be related to this series of changes and/or its preceding/following changes.

Knowing this, I cannot think of any workaround other than sticking to 6.6 in the meantime.

Thank you very much for the patch. Are there plans to also backport it to 2019?

[jaddie]

Just wanted to write to say I am so glad you have all written on here, I didn't even think about the fact I just upgraded my arch system, I was about to start tearing things apart this has saved me a heck of a lot of time, whilst I am here to say thank you, I can also confirm this is still happening on Arch Linux on 6.7.4

[massouji82]

Hi! We are running a msql based prosject on a mac and use the image mcr.microsoft.com/mssql/server:2019-latest through Podman. Podman will not start a container with this image since the kernel was updated. How kan we revert the kernel version of the host or is there another workaround? Any help would be highly appreciated. Thanks!

[johnvanham]

Same issue with Fedora 39 on 6.7.2 and 6.7.3, but fine on 6.6.x and 6.5.x (in case anyone is searching for this issue and using Fedora). Looking forward to the CU @fbrosseau

[zzzeek]

I think MSFT should strongly consider backporting this at least to SQL Server 2019 if not even 2017 as well. As people continue to upgrade their kernels this is going to be happening on an ever larger scale to existing SQL Server linux / container installations.

[kshpytsya]

Thank you very much for the patch. Are there plans to also backport it to 2019?

Am I missing something? I do not see any updated Docker images for mcr.microsoft.com/mssql/server:2022-latest that would make it run on 6.7.*.

[markbeazley]

Thank you very much for the patch. Are there plans to also backport it to 2019?
Am I missing something? I do not see any updated Docker images for mcr.microsoft.com/mssql/server:2022-latest that would make it run on 6.7.*.

It should be included in the next CU, no date estimate

The issue has been identified and should be fixed in the next SQL Server 2022 CU, but we cannot commit to a specific CU release or timeline as sometimes plans can unexpectedly change. No further investigation or data points should be needed for this, but thank you all for reporting and for looking for potential causes.

I've been keeping an eye on this page for a presumably CU12 to be released.

[kelkmere]

This problem just happens to my AWS EC2 Ubuntu 22.04 instance. I fixed it by downgrading the kernel from 6.8.0-1015-aws to 6.5.0-1014-aws. Here's the instruction: https://discourse.ubuntu.com/t/how-to-downgrade-the-kernel-on-ubuntu-20-04-to-the-5-4-lts-version/26459

[Illutax]

It happened also on my AWS EC2 Ubuntu 22.04.

We fixed it by updating the MSSQL Image from
mcr.microsoft.com/mssql/server:2019-CU13-ubuntu-20.04
to
mcr.microsoft.com/mssql/server:2019-CU28-ubuntu-20.04

[martin-schulze-e2m]

Are there any plans for a SQL Server 2019 release?

Looks like this has been fixed in 2019-CU26-ubuntu-20.04 release which is now 2019-latest.

Table here hasn't been updated here yet https://hub.docker.com/_/microsoft-mssql-server/ but checked the tag list and saw the new release (https://mcr.microsoft.com/v2/mssql/server/tags/list). All is working great now on macos, thank you!

I just found 2017-latest to be updated on 2024-09-10 in https://hub.docker.com/_/microsoft-mssql-server/ and it is working again.

[jcompagner]

this week i upgraded and restart my ubuntu 22.04.5 LTS and i noticed that the kernel did go from: 6.5.0-1024-aws to 6.8.0-1015-aws. After restart the sqlserver 2022 didn't work anymore
We did a pull of all the latest versions of that or specific CU's (like 12,13 or the latest) but none of them work.
But in these comments it is said that it is fixed but why do we still get:

`
This program has encountered a fatal error and cannot continue running at Wed Sep 25 20:20:18 2024
The following diagnostic information is available:

     Reason: 0x00000001
     Signal: SIGABRT - Aborted (6)
      Stack:
             IP               Function
             ---------------- --------------------------------------
             00005e170c63ca61 std::__1::bad_function_call::~bad_function_call()+0x97ab1
             00005e170c63c426 std::__1::bad_function_call::~bad_function_call()+0x97476
             00005e170c63b9af std::__1::bad_function_call::~bad_function_call()+0x969ff
             0000702c8cb12520 __sigaction+0x50
             0000702c8cb669fc pthread_kill+0x12c
             0000702c8cb12476 raise+0x16
             0000702c8caf87f3 abort+0xd3
             00005e170c63b88e std::__1::bad_function_call::~bad_function_call()+0x968de
             00005e170c5ce351 std::__1::bad_function_call::~bad_function_call()+0x293a1
             00005e170c5ced5d std::__1::bad_function_call::~bad_function_call()+0x29dad
             00005e170c59879b std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char>>::str() const+0xc7fb
             00005e170c57acfc <unknown>
             0000702c8caf9d90 __libc_init_first+0x90

`

for all releases of 2022 that we try to use?

[tobiajo]

What is causing the issue?

[danarj]

I'm having the same issue after upgrading the kernel to 6.8.0 now I don't have any work around

[Illutax]

What is causing the issue?

An Linux kernel update. Now older mssql docker images can't be run successfully. Try updating the docker image to an newer release.

[jcompagner]

What is causing the issue?

An Linux kernel update. Now older mssql docker images can't be run successfully. Try updating the docker image to an newer release.

But we tried the latest of a 2022 build , and none work anymore on our machine that runs ubuntu 2022.04 LTS with kernen 6.8

[005Dikshant]

This is happening for my local ubuntu 22.04.5 with kernel - Linux version 6.8.0-40-generic, when I am trying to build the docker image from the compose file, i am getting similar error -

this is my base image for sql server - FROM mcr.microsoft.com/mssql/server:2017-CU24-ubuntu-16.04

This program has encountered a fatal error and cannot continue running at Mon Sep 30 13:46:01 2024
mssql-prof` | The following diagnostic information is available:
mssql-prof |
mssql-prof | Reason: 0x00000001
mssql-prof | Signal: SIGABRT - Aborted (6)
mssql-prof | Stack:
mssql-prof | IP Function
mssql-prof | ---------------- --------------------------------------
mssql-prof | 00005f74251cbc35
mssql-prof | 00005f74251cb6c2
mssql-prof | 00005f74251cac11
mssql-prof | 00007af8dcb094c0 killpg+0x40
mssql-prof | 00007af8dcb09438 gsignal+0x38
mssql-prof | 00007af8dcb0b03a abort+0x16a
mssql-prof | 00005f7425131fed
mssql-prof | 00005f74251d7714
mssql-prof | 00005f74251fbd98
mssql-prof | 00005f74251fbb10
mssql-prof | 00005f74251408f4
mssql-prof | 00005f7425140489
mssql-prof | Process: 12 - sqlservr
mssql-prof | Thread: 84 (application thread 0x138)
mssql-prof | Instance Id: 252bbfb1-44a3-446a-b822-1c1f310459a2
mssql-prof | Crash Id: 1e0024a2-d6c5-499a-b853-4fbc68acb9e3
mssql-prof | Build stamp: 23b9dfd98170ce033c3334f80ad499e84f504fec8d90876c718146f04b79943f
mssql-prof | Distribution: Ubuntu 16.04.7 LTS
mssql-prof | Processors: 12
mssql-prof | Total Memory: 33463422976 bytes
mssql-prof | Timestamp: Mon Sep 30 13:46:01 2024
mssql-prof | Last errno: 2
mssql-prof | Last errno text: No such file or directory
mssql-prof | Executing: /opt/mssql/bin/handle-crash.sh with parameters
mssql-prof | handle-crash.sh
mssql-prof | /opt/mssql/bin/sqlservr
mssql-prof | 12
mssql-prof | /opt/mssql/bin
mssql-prof | /var/opt/mssql/log/
mssql-prof |
mssql-prof | 252bbfb1-44a3-446a-b822-1c1f310459a2
mssql-prof | 1e0024a2-d6c5-499a-b853-4fbc68acb9e3
mssql-prof |
mssql-prof |
mssql-prof | Ubuntu 16.04.7 LTS
mssql-prof | Capturing core dump and information to /var/opt/mssql/log...
mssql-prof | dmesg: read kernel buffer failed: Operation not permitted
mssql-prof | No journal files were found.
mssql-prof | No journal files were found.
mssql-prof | Mon Sep 30 13:46:01 UTC 2024 Capturing program information
mssql-prof | Mon Sep 30 13:46:02 UTC 2024 Attempting to capture a dump with paldumper for pid 12
mssql-prof | Captured a dump with paldumper
mssql-prof | Mon Sep 30 13:46:02 UTC 2024 Capturing program binaries
mssql-prof | Mon Sep 30 13:46:02 UTC 2024 Compressing the dump files

[Illutax]

This is happening for my local ubuntu 22.04.5 with kernel - Linux version 6.8.0-40-generic, when I am trying to build the docker image from the compose file, i am getting similar error -

this is my base image for sql server - FROM mcr.microsoft.com/mssql/server:2017-CU24-ubuntu-16.04

This program has encountered a fatal error and cannot continue running at Mon Sep 30 13:46:01 2024 mssql-prof` | The following diagnostic information is available: mssql-prof | mssql-prof | Reason: 0x00000001 mssql-prof | Signal: SIGABRT - Aborted (6) mssql-prof | Stack: mssql-prof | IP Function mssql-prof | ---------------- -------------------------------------- mssql-prof | 00005f74251cbc35 mssql-prof | 00005f74251cb6c2 mssql-prof | 00005f74251cac11 mssql-prof | 00007af8dcb094c0 killpg+0x40 mssql-prof | 00007af8dcb09438 gsignal+0x38 mssql-prof | 00007af8dcb0b03a abort+0x16a mssql-prof | 00005f7425131fed mssql-prof | 00005f74251d7714 mssql-prof | 00005f74251fbd98 mssql-prof | 00005f74251fbb10 mssql-prof | 00005f74251408f4 mssql-prof | 00005f7425140489 mssql-prof | Process: 12 - sqlservr mssql-prof | Thread: 84 (application thread 0x138) mssql-prof | Instance Id: 252bbfb1-44a3-446a-b822-1c1f310459a2 mssql-prof | Crash Id: 1e0024a2-d6c5-499a-b853-4fbc68acb9e3 mssql-prof | Build stamp: 23b9dfd98170ce033c3334f80ad499e84f504fec8d90876c718146f04b79943f mssql-prof | Distribution: Ubuntu 16.04.7 LTS mssql-prof | Processors: 12 mssql-prof | Total Memory: 33463422976 bytes mssql-prof | Timestamp: Mon Sep 30 13:46:01 2024 mssql-prof | Last errno: 2 mssql-prof | Last errno text: No such file or directory mssql-prof | Executing: /opt/mssql/bin/handle-crash.sh with parameters mssql-prof | handle-crash.sh mssql-prof | /opt/mssql/bin/sqlservr mssql-prof | 12 mssql-prof | /opt/mssql/bin mssql-prof | /var/opt/mssql/log/ mssql-prof | mssql-prof | 252bbfb1-44a3-446a-b822-1c1f310459a2 mssql-prof | 1e0024a2-d6c5-499a-b853-4fbc68acb9e3 mssql-prof | mssql-prof | mssql-prof | Ubuntu 16.04.7 LTS mssql-prof | Capturing core dump and information to /var/opt/mssql/log... mssql-prof | dmesg: read kernel buffer failed: Operation not permitted mssql-prof | No journal files were found. mssql-prof | No journal files were found. mssql-prof | Mon Sep 30 13:46:01 UTC 2024 Capturing program information mssql-prof | Mon Sep 30 13:46:02 UTC 2024 Attempting to capture a dump with paldumper for pid 12 mssql-prof | Captured a dump with paldumper mssql-prof | Mon Sep 30 13:46:02 UTC 2024 Capturing program binaries mssql-prof | Mon Sep 30 13:46:02 UTC 2024 Compressing the dump files

Try 2017-CU31-GDR2-ubuntu-18.04

[005Dikshant]

i downgrade from 6.8.0-40 generic to 6.5.0.41-generic now it is working again for my local system.

[alex-jitbit]

This is not an issue with Docker image, but a much bigger issue with MSSQL itself. Upgrading kernel on my AWS instances is now causing mssql 2017 to fail with this exact same error (under kernel 6.8.0). Booting to an older kernel (6.5.0-101`7) resolves the issue.