Persistent Environments

[ruffsl]

@matthieudelaro , nut looks like a neat project, just stumbled across it from you nvidia-docker issue. I'm interested in many of the same objectives you mention on the project readme. On a related note though, how do you feel about abusing volumes as a method to enable persistent development environments?

Perhaps is would take away from the portability or repeatably of the environments, but it would be convenient for when I need something like a persistent VM to tradition my user space apps from one linux LTS to the next, but would like to skip a hyper-visors or manual chroots.
https://forums.docker.com/t/persistent-development-environment-using-docker/9279

[matthieudelaro]

Hi @ruffsl

Thanks for sharing your ideas. Nut is young and new features are welcome.

I'm not sure I understand what you want to do though. Is any of the following related?

• you want to mount several directories in the container, which you can do by specifying several mount points in the nut.yml (as shown as Caffe example)
• you want to mount docker volumes in the container, which could be achieved with Nut by adding a new section in the nut.yml syntax (and by adapting the source code), such as:

syntax_version: "4"
project_name: caffe
based_on:
  docker_image: matthieudelaro/work-on-caffe
container_working_directory: /opt/caffe
mount:
  ...
macros:
  ...
volumes:  # new element of the syntax
  ... # define docker volumes to mount here

[ruffsl]

I'd like to mount an entire root linux file system used in the container, here is a sample script example I made to mount ever file in target directory as a folder in the root path. This was any system wide modifications made to the linux filesystem, like from an apt-get install, in the container could be preserved over container lifespans. This as opposed to repeatedly remembering to commit the container to an image, and flattening it if it gets too large.

By the way, how is nut's yaml file focuses and interests different from say a docker-compose yaml config?

[matthieudelaro]

By the way, how is nut's yaml file focuses and interests different from say a docker-compose yaml config?

While the Compose files are closely related to docker (you have to know how to use Docker in order to use compose files), the aim of Nut files is to describe what environment you want, even from the perspective of someone who doesn't know (much/anything) of docker. The philosophy is to define macros and request features (eg: GUI), not the unix sockets that you wish to expose to achieve this feature (eg: socket /tmp/.X11-unix). Nut takes care of this depending on the OS you are running, Docker Toolbox vs Docker for Mac, etc.

Now, you've probably noticed that version 4 doesn't respect this philosophy, if you consider that you have to specify working directory and mounting point. But at the beginning, the working directory and the mouting point were the current folder by default, so the only parameter about docker was the image you wished to use. This should be fixed by adding a parameter such as 'use_current_dir: yes, which would be equivalent tocontainer_working_directoryand propermount` parameter.

Another point: this simplistic philosophy for nut file addresses the need of users of environment (inheriting complex configuration from GitHub), but not the providers of environments (sharing their nut.yml on GitHub). Which is why nut files have to provide most configurations for Docker somehow.

As you and @kohlerm (issue #9) point out, it leads to a compose-like file / it could be implemented with a compose file. However, compose files do not declare macros or equivalent, which is the point of nut files. So the idea of inheriting from a compose file seems nice to me:

# nut.yml
# ...
based_on:
  compose:
    path: path/to/docker-compose.yml
    service: reddis  # name of service described in compose file
# ...

If the goal of using compose file is to generate a whole architecture of containers, I think this should happen in the container instead. Nut lunches a container based on an image name it has been provided, and you can do whatever you want in this container (compose, swarm, ...)

Thoughts? What do you think about this?

[kohlerm]

I solved this problem in my not yet open sourced pet project by assuming that there is always some kind of a top-level workspace directory
The toplevel workspace project would be determined by just searching the parent directories upwards and looking for a .git (or in my case also a .svn) directory. Alternatively an environment variable could be used to specify the top level workspace directory.
I would then make the workspace directory available in my config file (as a env variable) and then use that to correctly map the workspace directory into the container.

[kohlerm]

the idea about the compose file is that it has a well known syntax for defining parameters for running a container.
In my own project I did not yet implement compose support yet, but I came to the conclusion that it might be a good idea to specify parameters to run a container using compose instead of replicating it in my own config file.
Problem is that variable interpolation does not seem to be implemented in compose docker/compose#1377

which would be needed for what I described above.

So maybe compose could be used for defining default parameters and then you mostly override the volume parameters.

In any case emulating the compose syntax would be IMHO be good to lower the learning curve for people.

[matthieudelaro]

@ruffsl : support for docker volumes has been added in commit f8146fb

@kohlerm : from syntax V7, mount is replaced with volumes to get closer to docker-compose:

volumes:
  volume-example:
    volume_name: myvolume
    container_path: /mnt/myvolume
  equivalent-of-mount-in-V6:
    host_path: .
    container_path: /go/src/

Simply supporting docker-compose syntax is not trivial, because compose relies on docker client to run to start containers, whereas Nut calls docker host through docker api. This implies implementing all the nice features of docker cli (some of which are bottlenecks).

Moreover, compose have similar limitations as docker cli when it comes to paths for volumes (can't include a : in the path, even though it is a legal path). Nut doesn't have this issue.

Plus, declaring volumes and devices (devices are supported now) as a dictionary instead of a list is useful to override parent configuration.

[matthieudelaro]

@ruffsl : I updated the syntax so that you can specify a docker volume to manage persistent environment. I will close this issue now; feel free to reopen it if it doesn't fit your needs.