doc: syscall: document Setrlimit behavior change on Go 1.12 on macOS

[karalabe]

We've noticed a while back that when setting the file descriptor allowance on macos, Go restricts it to a lower value than requested:

package main

import (
	"fmt"
	"syscall"
)

func main() {
	// Get the current file descriptor limit
	var limit syscall.Rlimit
	if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
		panic(err)
	}
	// Try to update the limit to the max allowance
	limit.Cur = limit.Max
	if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
		panic(err)
	}
	// Try to get the limit again and see where it's at
	if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
		panic(err)
	}
	fmt.Println(limit.Cur)
}

The above code on Linux works as expected, setting the limit to the max returned by the syscall package.

On macos with Go 1.11.5 (and prior) it silently set it to the minimum of limit.Max and 10240. We had to patch our code to explicitly check the actual value set instead of relying on no-error-returned since we falsely alloted more fds to databases than available.

On macos with Go 1.12, our code again started to blow up, now returning invalid argument errors instead of the silent capping previously.

I'd probably say both behaviors are bad, but the new one is a lot worse than the previous. Now it only tells us it's invalid, but unless you know the magic number defined in the your own kernel version, there's no way for user code to figure out how many fds are actually available.

[ianlancetaylor]

CC @randall77

[randall77]

Yes, this is unfortunate. We switched to using the implementation of getrlimit in libSystem.dylib instead of raw system calls, and that's the behavior we get. In particular, the new implementation lies to us about the maximum value available.
I had to modify the TestRlimit with code like this:

	if runtime.GOOS == "darwin" && set.Cur > 10240 {
		// The max file limit is 10240, even though
		// the max returned by Getrlimit is 1<<63-1.
		// This is OPEN_MAX in sys/syslimits.h.
		set.Cur = 10240
	}

I suppose we could make our implementation of Getrlimit incorporate this fix internally. But my view of the syscall package is that you get what the OS provides, warts and all. And one day Apple may fix this issue (or change OPEN_MAX) and then any patch we have is obsolete.

You can always binary search for the largest allowable value. Yuck.

[randall77]

Note that this is not just a Go problem - C people run into similar issues:

https://postfix-devel.postfix.narkive.com/kVL334x7/mac-os-x-and-setrlimit-2

[karalabe]

Ah yes, I wasn't particularly complaining that Go is doing something wrong (given the restrictions), just trying to figure out what the best solution is. I guess my main complaint wasn't that Go works one way or the other, rather that it changed it behavior, breaking existing code.

We can fix it to handle both cases (i.e. hard code that magic number), just would like to know which solution will Go stay with in the end so we don't have to do a 3rd fixup round for Go 1.12.1 :)

[ianlancetaylor]

Going forward on Darwin Go is always going to call the C function.

[jeffallen]

I found that on Go 1.11.5, this code:

func init() {
	raiseFdLimit = func() {
		var rLimit syscall.Rlimit
		err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit)
		if err != nil {
			log.Fatal("Error Getting Rlimit ", err)
		}

		if rLimit.Cur < rLimit.Max {
			rLimit.Cur = rLimit.Max
			err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rLimit)
			if err != nil {
				log.Warn("Error Setting Rlimit:", err)
			}
		}

		err = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit)
		log.Info("File descriptor limit is:", rLimit.Cur)
	}
}

reports File descriptor limit is: 24576. On Go 1.12 it reports invalid argument. So I've added the workaround from above (#30401 (comment)) but instead of 10240, I've used 24576 and that returns the behavior of my server compiled with Go 1.12 to the same as it was on Go 1.11.5. (@randall77: MacOS version 10.14.3 (18D109))

The manpage says: setrlimit() now returns with errno set to EINVAL in places that historically succeeded. It no longer accepts "rlim_cur = RLIM_INFINITY" for RLIM_NOFILE. Use "rlim_cur = min(OPEN_MAX, rlim_max)".

I searched for OPEN_MAX in /Application/Xcode.app and found that it is 10240, even though the observed behavior is 24576 works. 🤷‍♂️

[randall77]

@jeffallen What Mac OS are you running?
I can't get ask for more than 10240 on Sierra (10.12.6) without getting an invalid argument error.

[jeffallen]

Here is an excellent explanation: https://unix.stackexchange.com/questions/350776/setting-os-x-macos-bsd-maxfile

It says this could be a problem on FreeBSD and OpenBSD as well.

Another useful reference, showing that reading the setrlimit manpage and paying any attention whatsoever to OPEN_MAX would be not a good idea: https://julio.meroh.net/2019/01/open-files-limit-macos-and-the-jvm.html

[bep]

For what it's worth;

On my Mojave 10.14.4 on Go 1.12.2, my tests tell me that I can set values in Setrlimit <= the value of limit as defined in /Library/LaunchDaemons/limit.maxfiles.plist (I assume that 10240 is the default?).

This can be set for the current session with something ala:

sudo launchctl limit maxfiles 1234 200000

In the above example, I get invalid argument for any argument to Setrlimit> 1234.

If I run Setrlimit as sudo, I can use any value.

[gopherbot]

Change https://golang.org/cl/188237 mentions this issue: doc/go1.13: document change in syscall.Setrlimit behavior

[gopherbot]

Change https://golang.org/cl/188357 mentions this issue: [release-branch.go1.12] doc/go1.12: document change in syscall.Setrlimit behavior

[frank-segui]

I found that on Go 1.11.5, this code:

func init() {
	raiseFdLimit = func() {
		var rLimit syscall.Rlimit
		err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit)
		if err != nil {
			log.Fatal("Error Getting Rlimit ", err)
		}

		if rLimit.Cur < rLimit.Max {
			rLimit.Cur = rLimit.Max
			err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rLimit)
			if err != nil {
				log.Warn("Error Setting Rlimit:", err)
			}
		}

		err = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit)
		log.Info("File descriptor limit is:", rLimit.Cur)
	}
}

reports File descriptor limit is: 24576. On Go 1.12 it reports invalid argument. So I've added the workaround from above (#30401 (comment)) but instead of 10240, I've used 24576 and that returns the behavior of my server compiled with Go 1.12 to the same as it was on Go 1.11.5. (@randall77: MacOS version 10.14.3 (18D109))

The manpage says: setrlimit() now returns with errno set to EINVAL in places that historically succeeded. It no longer accepts "rlim_cur = RLIM_INFINITY" for RLIM_NOFILE. Use "rlim_cur = min(OPEN_MAX, rlim_max)".

I searched for OPEN_MAX in /Application/Xcode.app and found that it is 10240, even though the observed behavior is 24576 works. 🤷‍♂️

I experienced the same. Found that sysctl kern.maxfilesperproc was set to 24576. I increased kern.maxfilesperproc and was able to increase rLimit.Cur to the new value.

This was in Mojave 10.14.6