Can't turn off WAL

[tlefevre]

Bug Report

The documentation is not clear on how to turn off DB.wal. I have tried Off, off and false. Nothing really works it seems?

Error posted on start up in stdout -> [2021/03/19 12:33:24] [error] [lib] backend failed.

There's nothing in the log file.

[nokute78]

WAL off feature will be supported from v1.7.3 , however it is not released yet. (latest version is v1.7.2)
https://docs.fluentbit.io/manual/pipeline/inputs/tail#wal-and-memory-usage

[tlefevre]

Ah ok, thanks! I didn't catch that.

"Note that WAL is not compatible with shared network file systems" - I need to turn it off, since we're using Fluent-bit in Kubernetes and storing the files on a GlusterFS mounted volume.

Reading the docs, it seems that it is not a good idea to use WAL.

[nokute78]

Thank you for comment. So, please be waiting next release.
Note: We supported this feature to fix #3073 memory resource issue.

I'm not familiar with WAL.
It is good information for me. Thank you.

https://sqlite.org/wal.html#concurrency

The wal-index greatly improves the performance of readers, but the use of shared memory means that all readers must exist on the same machine. This is why the write-ahead log implementation will not work on a network filesystem.

[tlefevre]

Do you know of anyone familiar with "why" it's a problem on shared network file systems?

We're so far just planning to run a single fluent-bit instance, so it might not be a problem at all even though it's on a Gluster filesystem.

[nokute78]

I don't know.

I think the hint is shared-memory. It is usually used for inter-process, not for inter-machine.
Refer to the sqlite document, wal-index is implemented using shared memory.
https://sqlite.org/wal.html#implementation_of_shared_memory_for_the_wal_index

[tlefevre]

Thanks for the help :)

[nokute78]

v1.7.3 has been released.
The option to handle WAL is db.journal_mode. (db.wal option is not supported)

We can set the following value.
#3239 (comment)