Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use mount instead of volume in docker compose file to avoid creating empty directories for missing files #473

Closed
mmguero opened this issue May 9, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@mmguero
Copy link
Collaborator

mmguero commented May 9, 2024

See Differences between -v and --mount behavior:

Because the -v and --volume flags have been a part of Docker for a long time, their behavior cannot be changed. This means that there is one behavior that is different between -v and --mount.

If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.

If you use --mount to bind-mount a file or directory that does not yet exist on the Docker host, Docker does not automatically create it for you, but generates an error.

Generating an error would be way preferable to creating an empty directory for our bind mounts.

As far as how to do this:

@mmguero mmguero added the enhancement New feature or request label May 9, 2024
@mmguero mmguero self-assigned this May 9, 2024
@mmguero mmguero added this to Malcolm May 9, 2024
@mmguero mmguero moved this to Todo (develop) in Malcolm May 9, 2024
@mmguero mmguero removed their assignment May 9, 2024
@mmguero mmguero modified the milestones: v24.05.0, v24.06.0 May 9, 2024
@mmguero mmguero self-assigned this May 20, 2024
@mmguero mmguero moved this from Todo (develop) to In Progress in Malcolm May 20, 2024
@mmguero
Copy link
Collaborator Author

mmguero commented May 20, 2024

As I have realized this is going to require some changes to install.py where we're adjusting the .yml, I'm going to bump this to a later release to handle it there.

I think the part we really want in the documentation is here: Volumes: Long Syntax

My rough-and-dirty attempt at find-and-replace in my text editor is (still requires some manual tweaking):

Find: ^( )(- *)([^:\n\r]+):([^:\n\r]+)(:\w+)?$
Replace: $1 - type: bind\n$1 bind:\n$1 create_host_path: false\n$1 source: $3\n$1 target: $4\n$1 read_only$5

in install.py the calls to ReplaceBindMountLocation are what do this, which is found here. Changing from short form to long form will break this, so we'll have to adjust that when we do it.

@mmguero mmguero moved this from In Progress to Todo (develop) in Malcolm May 20, 2024
@mmguero mmguero modified the milestones: v24.05.0, z.staging May 20, 2024
@mmguero mmguero modified the milestones: z.staging, v24.07.0 Jun 17, 2024
@mmguero mmguero moved this from Todo (develop) to In Progress in Malcolm Jul 15, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 15, 2024
…n the docker-compose.yml yet, this commit makes LocalPathForContainerBindMount work for when I do. it's backwards compatible with the old short-style mount options as well
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 15, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 16, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 16, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 16, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 17, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 17, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 17, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 18, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 18, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 18, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 18, 2024
@mmguero
Copy link
Collaborator Author

mmguero commented Jul 18, 2024

Testing now. Rather than create the empty directory (which is root-owned and often causes all sorts of headaches to clean up), now you get something like this:

Malcolm failed to start

 Network malcolm_default  Creating
 Network malcolm_default  Created
 Volume "malcolm_nginx-log-path"  Creating
 Volume "malcolm_nginx-log-path"  Created
 Container malcolm-suricata-1  Creating
 Container malcolm-htadmin-1  Creating
 Container malcolm-file-monitor-1  Creating
 Container malcolm-opensearch-1  Creating
 Container malcolm-upload-1  Creating
 Container malcolm-zeek-live-1  Creating
 Container malcolm-arkime-live-1  Creating
 Container malcolm-pcap-capture-1  Creating
 Container malcolm-suricata-live-1  Creating
 Container malcolm-filebeat-1  Creating
 Container malcolm-api-1  Creating
 Container malcolm-netbox-redis-1  Creating
 Container malcolm-netbox-postgres-1  Creating
 Container malcolm-freq-1  Creating
 Container malcolm-netbox-redis-cache-1  Creating
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: ./opensearch/opensearch.blahblah

All of the necessary files/directories SHOULD be created by install and auth_setup, but if for some reason they're not now we'll know it.

@mmguero mmguero moved this from In Progress to Testing in Malcolm Jul 18, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 18, 2024
@mmguero mmguero moved this from Testing to Done in Malcolm Jul 18, 2024
@mmguero mmguero closed this as completed Jul 18, 2024
This was referenced Jul 29, 2024
@mmguero mmguero moved this from Done to Released in Malcolm Jul 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Released
Development

No branches or pull requests

1 participant