mdbx_env_open() error "Invalid argument" on WSL (Windows Subsystem for Linux)

[oleg-kiriyenko]

Ubuntu 18.04/clang-9

   rc = mdbx_env_open(db->env, dir_path, 0, 0666);
    if (rc != MDBX_SUCCESS) {
        log_error(get_thread_data()->ctx, "mdbx_env_open() error: %s", mdbx_strerror(rc));

Got

mdbx_env_open() error: Invalid argument

But database files were created:

mdbx.dat  mdbx.lck

Corresponding part of strace output

mkdir("<path>", 0777) = -1 EEXIST (File exists)
openat(AT_FDCWD, "<path>/mdbx.dat", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 12
openat(AT_FDCWD, "<path>/mdbx.dat", O_WRONLY|O_DSYNC|O_CLOEXEC) = 13
openat(AT_FDCWD, "<path>/mdbx.lck", O_RDWR|O_CREAT|O_CLOEXEC, 0666) = 14
getpid()                                = 14076
sched_yield()                           = 0
fcntl(14, F_OFD_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=1}) = -1 EINVAL (Invalid argument)

[erthink]

fcntl(14, F_OFD_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=1})
   = -1 EINVAL (Invalid argument)

This is a valid fcntl() call if the kernel supports OFD-lock (3.15 and later).

Doest the make check succeed for libmdbx?

[oleg-kiriyenko]

make check also fails with the same error

[ 14319 child_1.1  fail ] [ 14322 child_4.4  fail ] mdbx_env_open() failed: Invalid argument (22)
[ 14320 child_2.2  fail ] [ 14327 child_9.9  fail ] mdbx_env_open() failed: Invalid argument (22)
[ 14325 child_7.7  fail ] [ 14324 child_6.6  fail ] [ 14326 child_8.8  fail ] mdbx_env_open() failed: Invalid argument (22)
mdbx_env_open() failed: Invalid argument (22)

The problem is in WSL.

Simila things here

microsoft/WSL#1927

[erthink]

The problem is in WSL.

Oh, WSL is mad and trouble full.
You may try WSL2 (but I am not sure) or native Windows-build of libmdbx, but should strongly avoid using linux-version of libmdbx with WSL.

For my part, I will try to add a check of WSL to prevent data loss.

[erthink]

@oleg-kiriyenko, could you clarify - does you using WSL or WSL2?

[oleg-kiriyenko]

Simple WSL. As far as I know WSL2 uses virtual machine which is not my case.

[erthink]

@oleg-kiriyenko, in fact you can use libmdbx on WSL with some restrictions:

• build libmdbx with the MDBX_USE_OFDLOCKS=0 option.
• simultaneously open DB from the one process only.

[AndreaLanfranchi]

Sorry for bringing this back to life but trying to build libmdbx with
#define MDBX_USE_OFDLOCKS 0
in config.h but after build the application reports panic

panic: fail to open mdbx: mdbx_env_open: no locks available, path: /tmp/tg-test-db626083220

I'm on WSL 2

[erthink]

@AndreaLanfranchi, libmdbx should run under WLS2 out-of-the box since no reasons against this.
Moreover it could run under WLS1 with some restrictions as noted above.
So it is unexpected that you have some problems with WSL2.
However, I can't check/debug this since I usen't Windows as a primary OS and can't run WSL2 VM inside Windows VM (i.e. via QEMU nor VirtualBox, etc).
Nonetheless, if you have problems, then I think they are not difficult to fix with your assistance.

[erthink]

The most likely causes of problems are:

1. Something is wrong with distinguishing WSL2 from WSL1, i.e. the

   libmdbx/src/lck-posix.c

   Lines 32 to 58 in 2b161db

   	static __cold uint8_t probe_for_WSL(const char *tag) {
   	const char *const WSL = strstr(tag, "WSL");
   	if (WSL && WSL[3] >= '2' && WSL[3] <= '9')
   	return WSL[3] - '0';
   	const char *const wsl = strstr(tag, "wsl");
   	if (wsl && wsl[3] >= '2' && wsl[3] <= '9')
   	return wsl[3] - '0';
   	return (WSL || wsl || strcasestr(tag, "Microsoft")) ? 1 : 0;
   	}
   	#endif /* Linux */
   	static __cold __attribute__((__constructor__)) void
   	mdbx_global_constructor(void) {
   	#if defined(__linux__) || defined(__gnu_linux__)
   	struct utsname buffer;
   	if (uname(&buffer) == 0) {
   	/* "Official" way of detecting WSL1 but not WSL2
   	* https://github.com/Microsoft/WSL/issues/423#issuecomment-221627364
   	*
   	* WARNING: False negative detection of WSL1 will result in DATA LOSS!
   	* So, the REQUIREMENTS for this code:
   	* 1. MUST detect WSL1 without false-negatives.
   	* 2. DESIRABLE detect WSL2 but without the risk of violating the first. */
   	mdbx_RunningOnWSL1 = probe_for_WSL(buffer.version) == 1 ||
   	probe_for_WSL(buffer.sysname) == 1 ||
   	probe_for_WSL(buffer.release) == 1;

   code is buggy or some important things were changes since Provide a way to positively detect WSL from an app compiled on Linux. microsoft/WSL#423 (comment).
   So please debug this code and (at least) show uname -a.
2. Something is wrong with the libmdbx build. Please show
   • output of mdbx_chk -V
   • output of mdbx_chk -vv path-to-your-db

[AndreaLanfranchi]

@erthink thank you very much for your response.

libmdbx should run under WLS2 out-of-the box since no reasons against this.
So it is unexpected that you have some problems with WSL2.

Unless I'm reading mistankenly this libmdbx detects WSL on behalf of this where basically the check is to look for either "Microsoft" or "WSLx" in the string reported by the equivalent linux command uname -r which in my case (I have a WSL2 environment with ubuntu 20.04) is 4.19.128-microsoft-standard even though my host environment reports

PS C:\Users\Andrea> wsl -l -v
  NAME            STATE           VERSION
* Ubuntu-20.04    Running         2

I'm on your side in embracing a safe position on panicing mdbx on detection of WSL (any) as I dug a lot on finding a proper way to detect WSL version. As far as I understand the linux kernels are the same for WSL1 and WSL2 and what differs is how the host (Windows 10 in this case) handles the virtualized env (in WSL2 there's a managed VM)

I'll report the output from tests you mentioned

[AndreaLanfranchi]

mdbx_chk -V

mdbx_chk version 0.10.0.27
 - source: v0.10.0-27-g2b161db6 2021-05-21T00:12:55+03:00, commit 2b161db6d8acf3fd5112972b09d5a1988a59ad8c, tree 41721da296c828dd3f7116a97f2ead5ccc38a107
 - anchor: dfe0671435fed2948a861ec25f1adbf99a470c88be69bcc27b32ca9a7ffc6787_v0_10_0_27_g2b161db6
 - build: 2021-05-24T16:59:20+0200 for x86_64-linux-gnu by cc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
 - flags: -std=gnu++2a -std=gnu11 -O2 -g -Wall -Werror -Wextra -Wpedantic -ffunction-sections -fPIC -fvisibility=hidden -pthread -Wno-error=attributes  -Wl,--gc-sections,-z,relro,-O1  -lrt
 - options: MDBX_DEBUG=0 MDBX_WORDBITS=64 BYTE_ORDER=LITTLE_ENDIAN MDBX_ENV_CHECKPID=AUTO=0 MDBX_TXN_CHECKOWNER=AUTO=1 MDBX_64BIT_ATOMIC=AUTO=1 MDBX_64BIT_CAS=AUTO=1 MDBX_TRUST_RTC=AUTO=0 MDBX_ENABLE_REFUND=1 MDBX_ENABLE_MADVISE=1 _GNU_SOURCE=YES MDBX_LOCKING=AUTO=2008 MDBX_USE_OFDLOCKS=AUTO=1 MDBX_CACHELINE_SIZE=64 MDBX_CPU_WRITEBACK_INCOHERENT=0 MDBX_MMAP_INCOHERENT_CPU_CACHE=0 MDBX_MMAP_INCOHERENT_FILE_WRITE=0 MDBX_UNALIGNED_OK=0 MDBX_PNL_ASCENDING=0

./mdbx_chk -vv

mdbx_chk v0.10.0-27-g2b161db6 (2021-05-21T00:12:55+03:00, T-41721da296c828dd3f7116a97f2ead5ccc38a107)
Running for /mnt/d/tg/tg/chaindata/ in 'read-only' mode...
 ! WSL1 (Windows Subsystem for Linux) is mad and trouble-full, injecting failure to avoid data loss, err 37
 ! mdbx_env_open() failed, error 37 No locks available

uname -a

Linux SKULL 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[AndreaLanfranchi]

This instead with #define MDBX_USE_OFDLOCKS 0 in config.h

mdbx_chk version 0.10.0.27
 - source: v0.10.0-27-g2b161db6 2021-05-21T00:12:55+03:00, commit 2b161db6d8acf3fd5112972b09d5a1988a59ad8c, tree 41721da296c828dd3f7116a97f2ead5ccc38a107
 - anchor: dfe0671435fed2948a861ec25f1adbf99a470c88be69bcc27b32ca9a7ffc6787_v0_10_0_27_g2b161db6
 - build: 2021-05-25T12:50:19+0200 for x86_64-linux-gnu by cc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
 - flags: -std=gnu++2a -std=gnu11 -O2 -g -Wall -Werror -Wextra -Wpedantic -ffunction-sections -fPIC -fvisibility=hidden -pthread -Wno-error=attributes  -Wl,--gc-sections,-z,relro,-O1  -lrt
 - options: MDBX_DEBUG=1 MDBX_WORDBITS=64 BYTE_ORDER=LITTLE_ENDIAN MDBX_ENV_CHECKPID=AUTO=0 MDBX_TXN_CHECKOWNER=AUTO=1 MDBX_64BIT_ATOMIC=AUTO=1 MDBX_64BIT_CAS=AUTO=1 MDBX_TRUST_RTC=AUTO=0 MDBX_ENABLE_REFUND=1 MDBX_ENABLE_MADVISE=1 MDBX_FORCE_ASSERTIONS=YES _GNU_SOURCE=YES MDBX_LOCKING=AUTO=2008 MDBX_USE_OFDLOCKS=0 MDBX_CACHELINE_SIZE=64 MDBX_CPU_WRITEBACK_INCOHERENT=0 MDBX_MMAP_INCOHERENT_CPU_CACHE=0 MDBX_MMAP_INCOHERENT_FILE_WRITE=0 MDBX_UNALIGNED_OK=0 MDBX_PNL_ASCENDING=0

./mdbx_chk -vv /mnt/d/tg/tg/chaindata/
mdbx_chk v0.10.0-27-g2b161db6 (2021-05-21T00:12:55+03:00, T-41721da296c828dd3f7116a97f2ead5ccc38a107)
Running for /mnt/d/tg/tg/chaindata/ in 'read-only' mode...
 ! WSL1 (Windows Subsystem for Linux) is mad and trouble-full, injecting failure to avoid data loss, err 37
 ! mdbx_env_open() failed, error 37 No locks available

[AndreaLanfranchi]

FYI I'm with turbo-geth team (now erigon)

[AndreaLanfranchi]

Should it be of any help is possible to invoke a windows command from within the linux shell.
Eg.

andrea@SKULL:~/erigon/ethdb/mdbx/dist$ cmd.exe /c wsl -l -v
'\\wsl$\Ubuntu-20.04\home\andrea\erigon\ethdb\mdbx\dist'
CMD.EXE è stato avviato utilizzando il percorso precedente come directory
corrente. I percorsi UNC non sono supportati. Per impostazione predefinita,
verrà utilizzata la directory di Windows.
  NAME            STATE           VERSION
* Ubuntu-20.04    Running         2

Apologies for italian output

[erthink]

Should it be of any help is possible to invoke a windows command from within the linux shell.

No-no, this is inappropriate way:

• such detection could be done only at runtime, but no during library build.
• but it would look crazy to try to call a windows program when initializing the linux library.

[erthink]

@AndreaLanfranchi, Please check the fix and either close the issue or I'll dig further (but later).

[AndreaLanfranchi]

@erthink thank you I will try ASAP.
Just skimmed over it and as I understand you rely on assumption kernel versions for WSL environment are increasing in WSL2 but is not expected on WSL1. Just wondering what happens on my machine if I revert back WSL from 2 to 1 (per Microsoft words there might be cases when WSL1 has to be preferred due to its faster filesystem integration).

[erthink]

Just skimmed over it and as I understand you rely on assumption kernel versions for WSL environment are increasing in WSL2 but is not expected on WSL1. Just wondering what happens on my machine if I revert back WSL from 2 to 1 (per Microsoft words there might be cases when WSL1 has to be preferred due to its faster filesystem integration).

AFAIK, in this case you're will get the 4.4.x kernel back.
Anyway, the assumption on kernel version looks for me better than an assumption on the case of the first letter of Microsoft ;)

[AndreaLanfranchi]

So without #define MDBX_USE_OFDLOCKS 0

mdbx_chk v0.10.0-28-gbcc546b (2021-05-25T15:18:15+03:00, T-67c9b2b0bd564ee1554b556db26fd49bd5ede53a)
Running for /mnt/d/tg/tg/chaindata/ in 'read-only' mode...
 ! mdbx_env_open() failed, error 22 Invalid argument

While instead with #define MDBX_USE_OFDLOCKS 0

andrea@SKULL:~/libmdbx/dist$ ./mdbx_chk -vv /mnt/d/tg/tg/chaindata/
mdbx_chk v0.10.0-28-gbcc546b (2021-05-25T15:18:15+03:00, T-67c9b2b0bd564ee1554b556db26fd49bd5ede53a)
Running for /mnt/d/tg/tg/chaindata/ in 'read-only' mode...
   open-MADV_DONTNEED 211640009..211812352
   readahead OFF 0..211640009
 - monopolistic mode
 - current boot-id 9fc9222a1ff58f53-0be4819d875e496c
 - pagesize 4096 (4096 system), max keysize 1980..2022, max readers 118
 - mapsize 2199023255552 (2.00 Tb)
 - dynamic datafile: 12288 (12.00 Kb) .. 2199023255552 (2.00 Tb), +2147483648 (2.00 Gb), -4294967296 (4.00 Gb)
 - current datafile: 867583393792 (808.00 Gb), 211812352 pages
 - transactions: recent 221, latter reader 221, lag 0
 - meta-0: steady txn#220, stay
 - meta-1: steady txn#221, head
 - meta-2: steady txn#219, tail
 - performs check for meta-pages clashes
 - performs full check recent-txn-id with meta-pages
Traversal b-tree by txn#221...
 - found 'AccountChangeSet' area
^C - interrupted by signal
 - pages: walked 1477430, left/unused 210162579
     @GC: subtotal 5428, large 2, leaf 1
     @MAIN: subtotal 1, leaf 1
     @META: subtotal 3
     AccountChangeSet: subtotal 1471998, branch 187575, leaf 1284423
 - usage: total 6051553280 bytes, payload 4303186031 (71.1%), unused 1748367249 (28.9%)
 - summary: average fill 71.1%, 0 problems
Processing '@MAIN'...
 - key-value kind: usual-key => single-value, flags: none
 - page size 4096, entries 48
 - b-tree depth 1, pages: branch 0, leaf 1, overflow 0
 - summary: 0 records, 0 dups, 0 key's bytes, 0 data's bytes, 0 problems
Processing '@GC'...
 - key-value kind: ordinal-key => single-value, flags: integerkey
 - last modification txn#221
 - page size 4096, entries 2
 - b-tree depth 1, pages: branch 0, leaf 1, overflow 5427
 - summary: 0 records, 0 dups, 0 key's bytes, 0 data's bytes, 0 problems
 - space: 536870912 total pages, backed 211812352 (39.5%), allocated 211640009 (39.4%), remained 325230903 (60.6%), used 1477430 (0.3%), gc 0 (0.0%), detained 0 (0.0%), reclaimable 0 (0.0%), available 325230903 (60.6%)
No error is detected, elapsed 305.085 seconds

I interrupted it as the db is huge

[erthink]

@AndreaLanfranchi, please show cat /proc/mounts for your /mnt/d.

[AndreaLanfranchi]

Here are the Windows partitions auto mounted by WSL2 in linux environment

C:\134 /mnt/c 9p rw,dirsync,noatime,aname=drvfs;path=C:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=8,wfd=8 0 0
D:\134 /mnt/d 9p rw,dirsync,noatime,aname=drvfs;path=D:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=8,wfd=8 0 0
E:\134 /mnt/e 9p rw,dirsync,noatime,aname=drvfs;path=E:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=8,wfd=8 0 0
F:\134 /mnt/f 9p rw,dirsync,noatime,aname=drvfs;path=F:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=8,wfd=8 0 0

[erthink]

I need to know the filesystem type visible from inside WSL2 to add it to the mdbx_check_fs_local()'s black list.

[AndreaLanfranchi]

It is drvfs

[AndreaLanfranchi]

andrea@SKULL:/mnt/c/Users/Andrea$ findmnt
TARGET                          SOURCE       FSTYPE      OPTIONS
/                               /dev/sdb     ext4        rw,relatime,discard,errors=remount-ro,data=ordered
├─/mnt/wsl                      tmpfs        tmpfs       rw,relatime
├─/init                         tools[/init] 9p          ro,relatime,dirsync,aname=tools;fmask=022,loose,access=client,trans=fd,rfd=6,wfd=6
├─/dev                          none         devtmpfs    rw,nosuid,relatime,size=13103444k,nr_inodes=3275861,mode=755
│ └─/dev/pts                    devpts       devpts      rw,nosuid,noexec,noatime,gid=5,mode=620,ptmxmode=000
├─/sys                          sysfs        sysfs       rw,nosuid,nodev,noexec,noatime
│ └─/sys/fs/cgroup              tmpfs        tmpfs       rw,nosuid,nodev,noexec,relatime,mode=755
│   ├─/sys/fs/cgroup/unified    cgroup2      cgroup2     rw,nosuid,nodev,noexec,relatime,nsdelegate
│   ├─/sys/fs/cgroup/cpuset     cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,cpuset
│   ├─/sys/fs/cgroup/cpu        cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,cpu
│   ├─/sys/fs/cgroup/cpuacct    cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,cpuacct
│   ├─/sys/fs/cgroup/blkio      cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,blkio
│   ├─/sys/fs/cgroup/memory     cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,memory
│   ├─/sys/fs/cgroup/devices    cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,devices
│   ├─/sys/fs/cgroup/freezer    cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,freezer
│   ├─/sys/fs/cgroup/net_cls    cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,net_cls
│   ├─/sys/fs/cgroup/perf_event cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,perf_event
│   ├─/sys/fs/cgroup/net_prio   cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,net_prio
│   ├─/sys/fs/cgroup/hugetlb    cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,hugetlb
│   ├─/sys/fs/cgroup/pids       cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,pids
│   └─/sys/fs/cgroup/rdma       cgroup       cgroup      rw,nosuid,nodev,noexec,relatime,rdma
├─/proc                         proc         proc        rw,nosuid,nodev,noexec,noatime
│ └─/proc/sys/fs/binfmt_misc    binfmt_misc  binfmt_misc rw,relatime
├─/run                          none         tmpfs       rw,nosuid,noexec,noatime,mode=755
│ ├─/run/lock                   none         tmpfs       rw,nosuid,nodev,noexec,noatime
│ ├─/run/shm                    none         tmpfs       rw,nosuid,nodev,noatime
│ └─/run/user                   none         tmpfs       rw,nosuid,nodev,noexec,noatime,mode=755
├─/mnt/c                        C:\          9p          rw,noatime,dirsync,aname=drvfs;path=C:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=8,wfd=8
├─/mnt/d                        D:\          9p          rw,noatime,dirsync,aname=drvfs;path=D:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=8,wfd=8
├─/mnt/e                        E:\          9p          rw,noatime,dirsync,aname=drvfs;path=E:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=8,wfd=8
└─/mnt/f                        F:\          9p          rw,noatime,dirsync,aname=drvfs;path=F:\;uid=1000;gid=1000;symlinkroot=/mnt/,mmap,access=client,msize=65536,trans=fd,rfd=8,wfd=8

[AndreaLanfranchi]

MS doc about DrvFS

[erthink]

Thanks.
In general, the problem is devfs:

1. Actually it is a remote filesystem from Linux's kernel point of view.
   The contents of the database cannot be coherent for the processes that opened it on the side of the remote host (Windows) and the local system (Linux).
   Therefore, such database can only be opened by single process in exclusive mode.
2. Devfs (like VolFs) has a lot of limitations and in particular does not support OFD-locks for files.

The problem with OFD-locks (probably) can be workarounded, but I'm not at all sure Devfs will work stably and will not destroy your data.
So you shouldn't open inside WSL2 a database hosted in Windows (aka "Don't eat with your knife.").

[AndreaLanfranchi]

Thank you. All your comments make perfectly sense.
One of the requirements of Erigon is to run on Windows too and up until last week all build tools were designed only for *nix system thus the need to have a build on WSL pointing data to other windows partitions (linux filesystem of WSL machine is - by default - in Windows' system partition and in 99% of cases there's not enough space to store or 1TB+ db).
Actually we have a PowerShell script which builds the whole solution natively on windows so the requirement of WSL is less important.

[erthink]

After consulting with colleagues using WSL2, I decided to allow a database to be opened on 9P/WSL2 filesystem in exclusive mode.
They claim that it should work stable.

So you can use the mounted disk if you add the MDBX_EXCLUSIVE option when opening the database.

[AndreaLanfranchi]

Thank you.
Just tested your latest commits and they're working fine without the need to add #define MDBX_USE_OFDLOCKS 0
Do you have an ETA for this devel branch merged into master and release ?

[erthink]

Do you have an ETA for this devel branch merged into master and release ?

Today.