Issues with mongo with mounted volume for data #28

skyksandr · 2015-12-11T14:22:17Z

I've created container with MongoDB.
Dockerfile:

FROM centos:centos6.7
MAINTAINER Aleksandr Kunin <[email protected]>

RUN yum -y update \
    && yum -y install epel-release \
    && yum clean all

RUN yum -y install mongodb-server

RUN mkdir -p /data/db

EXPOSE 27017
ENTRYPOINT ["/usr/bin/mongod", "--smallfiles"]

docker-compose.yml

db:
  image: qstnnr/mongodb
  volumes:
    - /data/db:/data/db

I have two problems with it:

1. If I put in docker-compose.yml OS X folder

    - ./db:/data/db

Container do not start with error:
Assertion: 13651:Couldn't fsync directory '/data/db': errno:22 Invalid argument

What I found on this topic is http://stackoverflow.com/questions/29570989/docker-mongodb-share-volume-with-mac-os-x:

It seems to be specific to the mongo software and the use of virtualbox , see the README

WARNING: because MongoDB uses memory mapped files it is not possible to use it through vboxsf to your host vbox bug.
It doesn't behave like that if you keep the datadir out of the mounted volume.

2. If I put in docker-compose.yml folder from my docker-machine VM

 - /data/db:/data/db

Container starts but do not finishes. In result I have to reboot docker-machine VM because nothing can kill mongod process.

The text was updated successfully, but these errors were encountered:

legal90 · 2015-12-13T09:35:08Z

I should to clarify some important details about Docker volumes and shared folders:

While using Docker Machine, it is assumed that the "Docker Host" is a VM, not a Mac host. It means that when we run docker (or docker-compose) on Mac, all volume paths are located on the VM. In your example, /data/db (on the left side of the colon) is the VM's path which will be mapped to the container. Docker will create this path if it does not exist.
For the convenience, local drivers for Docker Machine (as well as virtualbox and parallels) mount the whole /Users folder from your Mac host to the /Users path in each Docker Machine VM. So, your /Users content became available from the VM by the same path and could be mapped to containers.

So, if you set the volume path which is a child of /Users (for example, relative path ./db), then the Mac's folder will be mapped to container. In this case the filesystem is prl_fs or vboxsf, depending on the driver.
If you set some other volume path (/data/db), then this dir will be created on the VM only and it will not be accessible from your Mac directly. In this case there are no shared folders involved.

legal90 · 2015-12-13T09:35:11Z

Let's abstract away from the Compose use case, there is an easier way to reproduce the subject issue. And yes, seems like Parallels Shared Folders could not be used for Mongo DB storage at this moment:

# We assume that Parallels Desktop VM is created and env is configured:
# eval "$(docker-machine env your-vm-name)"

$ docker run -v ~/test-docker-volume/data/db:/data/db -p 27017:27017 mongo --smallfiles

2015-12-13T08:50:55.589+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=6d735a8a9892
2015-12-13T08:50:55.590+0000 I CONTROL  [initandlisten] db version v3.2.0
2015-12-13T08:50:55.590+0000 I CONTROL  [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
2015-12-13T08:50:55.591+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2015-12-13T08:50:55.591+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2015-12-13T08:50:55.591+0000 I CONTROL  [initandlisten] modules: none
2015-12-13T08:50:55.591+0000 I CONTROL  [initandlisten] build environment:
2015-12-13T08:50:55.592+0000 I CONTROL  [initandlisten]     distmod: debian71
2015-12-13T08:50:55.592+0000 I CONTROL  [initandlisten]     distarch: x86_64
2015-12-13T08:50:55.592+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2015-12-13T08:50:55.592+0000 I CONTROL  [initandlisten] options: { storage: { mmapv1: { smallFiles: true } } }
2015-12-13T08:50:55.598+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-12-13T08:50:55.608+0000 E STORAGE  [initandlisten] WiredTiger (1) [1449996655:608691][1:0x7f33ade64c80], connection: /data/db/WiredTiger.wt: Operation not permitted
2015-12-13T08:50:55.610+0000 I -        [initandlisten] Assertion: 28595:1: Operation not permitted
2015-12-13T08:50:55.622+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 1: Operation not permitted, terminating
2015-12-13T08:50:55.623+0000 I CONTROL  [initandlisten] dbexit:  rc: 100

Seems like our prl_fs filesystem returns "Operation not permitted" for wiredtiger_open call. @romankulikov, is it similar to Virtualbox ticket about mmapped access? (https://www.virtualbox.org/ticket/819).

It works fine on the same VM, outside of Parallels Shared Folders:

$ docker run -v /data/db:/data/db -p 27017:27017 mongo --smallfiles

2015-12-13T09:20:54.234+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=de303b29f580
2015-12-13T09:20:54.235+0000 I CONTROL  [initandlisten] db version v3.2.0
2015-12-13T09:20:54.235+0000 I CONTROL  [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
2015-12-13T09:20:54.235+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2015-12-13T09:20:54.235+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2015-12-13T09:20:54.236+0000 I CONTROL  [initandlisten] modules: none
2015-12-13T09:20:54.236+0000 I CONTROL  [initandlisten] build environment:
2015-12-13T09:20:54.236+0000 I CONTROL  [initandlisten]     distmod: debian71
2015-12-13T09:20:54.236+0000 I CONTROL  [initandlisten]     distarch: x86_64
2015-12-13T09:20:54.237+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2015-12-13T09:20:54.237+0000 I CONTROL  [initandlisten] options: { storage: { mmapv1: { smallFiles: true } } }
2015-12-13T09:20:54.240+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-12-13T09:20:54.280+0000 W STORAGE  [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger
2015-12-13T09:20:54.280+0000 I CONTROL  [initandlisten]
2015-12-13T09:20:54.280+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten]
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten]
2015-12-13T09:20:54.287+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2015-12-13T09:20:54.289+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2015-12-13T09:20:54.289+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker

romankulikov · 2015-12-28T16:11:50Z

@legal90, I do think this is the same problem. prl_fs supports mmap-ed files, but with significant restrictions. For example, opening mmap-ed files for read-write is prohibited. Seems we should consider this to be improved from our side.

alevlasu · 2018-08-29T11:55:28Z

It needs to be checked against Parallels Desktop 14.

legal90 · 2020-04-26T18:50:15Z

The issue still persist on Parallels Desktop 15.1.3.

With image mongo:3.2.0, the same what we tried above 4 years ago:

$ docker run -v $(pwd)/:/data/db -p 27017:27017 mongo:3.2.0

2020-04-26T18:42:48.998+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=86c88cc84166
2020-04-26T18:42:48.998+0000 I CONTROL  [initandlisten] db version v3.2.0
2020-04-26T18:42:48.999+0000 I CONTROL  [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
2020-04-26T18:42:48.999+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2020-04-26T18:42:48.999+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-04-26T18:42:48.999+0000 I CONTROL  [initandlisten] modules: none
2020-04-26T18:42:48.999+0000 I CONTROL  [initandlisten] build environment:
2020-04-26T18:42:48.999+0000 I CONTROL  [initandlisten]     distmod: debian71
2020-04-26T18:42:48.999+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-04-26T18:42:48.999+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-04-26T18:42:49.000+0000 I CONTROL  [initandlisten] options: {}
2020-04-26T18:42:49.003+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2020-04-26T18:42:49.010+0000 E STORAGE  [initandlisten] WiredTiger (17) [1587926569:10438][1:0x7ff458c55c80], connection: /data/db/WiredTiger.wt: File exists
2020-04-26T18:42:49.010+0000 I STORAGE  [initandlisten] WiredTiger unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1
2020-04-26T18:42:49.011+0000 E STORAGE  [initandlisten] WiredTiger (1) [1587926569:11385][1:0x7ff458c55c80], connection: /data/db/WiredTiger.wt: Operation not permitted
2020-04-26T18:42:49.012+0000 I -        [initandlisten] Assertion: 28595:1: Operation not permitted
2020-04-26T18:42:49.012+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 1: Operation not permitted, terminating
2020-04-26T18:42:49.012+0000 I CONTROL  [initandlisten] dbexit:  rc: 100

With image mongo:4.2.6, the latest at this moment:

$ docker run -v $(pwd)/:/data/db -p 27017:27017 mongo:4.2.6

2020-04-26T18:47:39.495+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-04-26T18:47:39.497+0000 W  ASIO     [main] No TransportLayer configured during NetworkInterface startup
2020-04-26T18:47:39.498+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=e6fffb9170e6
2020-04-26T18:47:39.498+0000 I  CONTROL  [initandlisten] db version v4.2.6
2020-04-26T18:47:39.498+0000 I  CONTROL  [initandlisten] git version: 20364840b8f1af16917e4c23c1b5f5efd8b352f8
2020-04-26T18:47:39.498+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
2020-04-26T18:47:39.498+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2020-04-26T18:47:39.498+0000 I  CONTROL  [initandlisten] modules: none
2020-04-26T18:47:39.498+0000 I  CONTROL  [initandlisten] build environment:
2020-04-26T18:47:39.498+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
2020-04-26T18:47:39.499+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2020-04-26T18:47:39.499+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2020-04-26T18:47:39.499+0000 I  CONTROL  [initandlisten] options: { net: { bindIp: "*" } }
2020-04-26T18:47:39.500+0000 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2020-04-26T18:47:39.893+0000 E  STORAGE  [initandlisten] WiredTiger error (17) [1587926859:893216][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:893216][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists
2020-04-26T18:47:39.894+0000 I  STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.7
2020-04-26T18:47:39.895+0000 E  STORAGE  [initandlisten] WiredTiger error (1) [1587926859:895416][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:895416][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted
2020-04-26T18:47:39.901+0000 E  STORAGE  [initandlisten] WiredTiger error (17) [1587926859:901779][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:901779][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists
2020-04-26T18:47:39.903+0000 I  STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.8
2020-04-26T18:47:39.903+0000 E  STORAGE  [initandlisten] WiredTiger error (1) [1587926859:903528][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:903528][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted
2020-04-26T18:47:39.909+0000 E  STORAGE  [initandlisten] WiredTiger error (17) [1587926859:909346][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:909346][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists
2020-04-26T18:47:39.910+0000 I  STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.9
2020-04-26T18:47:39.910+0000 E  STORAGE  [initandlisten] WiredTiger error (1) [1587926859:910614][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:910614][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted
2020-04-26T18:47:39.917+0000 E  STORAGE  [initandlisten] WiredTiger error (17) [1587926859:917648][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:917648][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists
2020-04-26T18:47:39.918+0000 I  STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.10
2020-04-26T18:47:39.918+0000 E  STORAGE  [initandlisten] WiredTiger error (1) [1587926859:918737][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:918737][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted
2020-04-26T18:47:39.924+0000 E  STORAGE  [initandlisten] WiredTiger error (17) [1587926859:924934][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:924934][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists
2020-04-26T18:47:39.925+0000 I  STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.11
2020-04-26T18:47:39.926+0000 E  STORAGE  [initandlisten] WiredTiger error (1) [1587926859:926055][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:926055][1:0x7f236ff77b00], connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted
2020-04-26T18:47:39.927+0000 W  STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.
2020-04-26T18:47:39.928+0000 F  STORAGE  [initandlisten] Reason: 1: Operation not permitted
2020-04-26T18:47:39.928+0000 F  -        [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 915
2020-04-26T18:47:39.928+0000 F  -        [initandlisten]

***aborting after fassert() failure

Although, it's pretty rear use case. I'm not sure how often people want to mount the db dir to the container from their Mac host 🤔

romankulikov · 2020-05-06T05:47:18Z

The issue still persist on Parallels Desktop 15.1.3.

Investigating this. The thing is that WiredTiger starts successfully over SharedFolders on Ubuntu distro, but does fail in boot2docker.

teran · 2021-05-07T10:08:13Z

My +1 for this issue.
I've noticed kinda similar behaviour with PostgreSQL database on external volume mounted from the host.

Any updates on that?

romankulikov · 2021-05-07T13:00:17Z

I've failed to debug the problem the previous year and postponed an attempt. If there's an interest in the problem I may try again. Any additional info and steps to reproduce are welcome.

teran · 2021-05-07T13:07:17Z

Hello @romankulikov,

Thanks for the update!

I've made a really trivial thing: launched docker run -v /Volumes/externalDisk/directory:/data postgres and tried to run initdb on that directory.

Things I've noted:

chown -R postgres:postgres /data made no changes on source filesystem, i.e. have no changed owner UID:GID on /Volumes/externalDisk/directory so it looks like there's some mapping should be within the driver or it's not supported at all
initdb failed on creating directory for wal so it looks like mkdir was failed in my case.

I could gather more information including strace output if needed but it looks like it should be reproduced in really simple way.

BTW. RO-volume is working perfectly so it looks like it's only about writes.

Please let let me know if you need any further details.

legal90 self-assigned this Dec 11, 2015

legal90 assigned romankulikov and unassigned legal90 Jan 4, 2016

legal90 mentioned this issue Jan 4, 2016

Parallels driver not syncing folder #25

Closed

legal90 added the upstream/parallels label Mar 30, 2016

romankulikov mentioned this issue Aug 20, 2016

prl_fs doesn't support wine: NtMapViewOfSection map_file_into_view #52

Closed

seantanly mentioned this issue Nov 30, 2017

Postgresql problem with mounted volume for data #71

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues with mongo with mounted volume for data #28

Issues with mongo with mounted volume for data #28

skyksandr commented Dec 11, 2015

legal90 commented Dec 13, 2015

legal90 commented Dec 13, 2015

romankulikov commented Dec 28, 2015

alevlasu commented Aug 29, 2018

legal90 commented Apr 26, 2020

romankulikov commented May 6, 2020

teran commented May 7, 2021

romankulikov commented May 7, 2021

teran commented May 7, 2021

Issues with mongo with mounted volume for data #28

Issues with mongo with mounted volume for data #28

Comments

skyksandr commented Dec 11, 2015

1. If I put in docker-compose.yml OS X folder

2. If I put in docker-compose.yml folder from my docker-machine VM

legal90 commented Dec 13, 2015

legal90 commented Dec 13, 2015

romankulikov commented Dec 28, 2015

alevlasu commented Aug 29, 2018

legal90 commented Apr 26, 2020

romankulikov commented May 6, 2020

teran commented May 7, 2021

romankulikov commented May 7, 2021

teran commented May 7, 2021