Call to mkstemp() failed with Invalid argument (error 22) (was: Unable to start 3-nodes cluster)

[rystsov]

Hi folks,

I'm trying to start a 3-nodes cluster and yb-master fails with the following error:

Log file created at: 2018/01/23 07:19:33
Running on machine: yuga3
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0123 07:19:33.281553    44 master_main.cc:82] Check failed: _s.ok() Bad status: IO error (yb/util/env_posix.cc:202): Unable to initialize catalog manager: Failed to initialize sys tables async: Failed to open new log: Call to mkstemp() failed on name template /yuga/mem/yb-master/yb-data/master/wals/table-sys.catalog.uuid/tablet-00000000000000000000000000000000/.tmp.newsegmentXXXXXX: Invalid argument (error 22)

Environment
OS: Ubuntu 17.10
Docker: 17.12.0-ce
Host: macOS 10.13.2
YugaByte: 0.9.1.0

I followed this instructions to install YugaByte and then this page to start a cluster but I skipped the Prerequisites steps since they are CentOS 7 specific and I use Ubuntu.

Containers has the following mapping between hostnames and IPs:

yuga1: 172.31.0.4
yuga2: 172.31.0.2
yuga3: 172.31.0.3

On each container I executed the following command ($yuga* was set to a corresponding IP address):

/yuga/yugabyte-0.9.1.0/bin/yb-master --master_addresses $yuga1:7100,$yuga2:7100,$yuga3:7100 --fs_data_dirs "/yuga/mem/yb-master"

On yuga3 the command failed with the mentioned error. Please find the whole data dirs attached (including the logs).

[ramkumarvs]

@rystsov Thanks for getting started with YugaByte db. Have few questions on your setup

• I assume you are bringing 3 docker containers of ubuntu and copied the yugabyte binary there and start those as three node clusters?
• Start yb-master individually inside of each of those containers.

[rystsov]

@ramkumarvs yes, you're right, I started three containers in the same bridge-network, all containers were in privileged mode and then I manually started yb-master in each of them

[bmatican]

Hey @rystsov ! Doing some more debugging on our end, would you mind sharing what file system you're pointing the yb-master to and if you could explicitly check if it supports O_DIRECT:
xfs_io -f -c "pwrite 0 20k" /yuga/mem/test1 -d (might need xfsprogs for this)

[rystsov]

@bmatican I don't have access to that information right now, I'll update it in the evening, but I used a docker compose file very similar to this one, and /yuga/mem was Docker's tmpfs.

By the way, I observed the issue only on the last node I run yb-master. If I run it first on yuga1 and yuga2 then the issue was on yuga3; if I run it first on yuga2 and yuga3 then the issue was with yuga1.

[bmatican]

Ok, after a bit more investigation, seems like it is indeed a FS issue, that O_DIRECT does not work on tmpfs. To bypass this, @ramkumarvs tested bringing up the masters with --fs_data_dirs pointing to a non-tmpfs path inside the containers and it seemed to work just fine.

Also, based on docker/for-mac#1619 and our own testing, it seems there might be an issue with mac based host mounted volumes as well, where O_DIRECT would not work there either.

I've separately filed #18 to address the underlying issue of not crashing if we find ourselves in this case, but that's more of a longer term fix.

For a quick term solution on your end:

• do not use tmpfs for the location for fs_data_dirs
• do not use a host mounted volume for the location of fs_data_dirs

Hope that helps!

[kmuthukk]

Will close this issue out; hope that suggested workaround of using a non "tmpfs" directory is sufficient in the interim. As @bmatican wrote, we have file #18 to handle this case gracefully.