diff --git a/README.md b/README.md index b372da7..ed5f41c 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,9 @@ -[![Actions Status](https://github.com/Stratoscale/skipper/workflows/build/badge.svg)](https://github.com/Stratoscale/skipper/actions) # Skipper +## Easily dockerize your Git repository + +[![Build Status](https://github.com/Stratoscale/skipper/workflows/build/badge.svg?branch=upstream)](https://github.com/Stratoscale/skipper/actions) [![Release Status](https://github.com/Stratoscale/skipper/workflows/Releases/badge.svg)](https://github.com/Stratoscale/skipper/actions) [![PyPI - Version](https://img.shields.io/pypi/v/strato-skipper?logo=pypi&label=pypi%20package&color=2334D058&cacheSeconds=10)](https://pypi.org/project/strato-skipper) + ## Introduction Use Skipper to build & test your project in an isolated environment, using Docker containers with pre-defined sane configuration. @@ -9,31 +12,49 @@ Skipper allows you to execute makefile targets inside a container (or just run a ## Installation It is recommended to install Skipper directly from PyPi: -```bash + +```shell sudo pip install strato-skipper -``` +``` You can also install Skipper from source: -``` bash + +``` shell git clone http://github.com/Stratoscale/skipper sudo make install ``` Configure bash completion for skipper by sourcing the completion script in your ~/.bashrc file: + ``` bash echo 'source <(skipper completion)' >>~/.bashrc ``` ## Python3 Environment + Skipper supports building and running in Python3 environment Set your locale to UTF-8: -```bash + +```shell export LC_ALL="en_US.UTF-8" export LANG="en_US.UTF-8" ``` + +## Note for Linux Users with Docker + +If you run on Linux and use Docker without sudo, Skipper will create a dedicated user inside the build container with both root and docker groups. Commands are executed on behalf of this user. + +To preserve the environment (e.g., PATH), Skipper uses the `su` command with the `-m` flag. However, on Debian distros, even with the `-m` flag specified, the PATH variable may be reset. As a workaround, Skipper attempts to use `sudo -sE` (if installed) as an alternative to maintain the environment. + +If you prefer to use sudo, please install it in the build container. Additionally, it is required to disable `env_reset` with `secure_path` in `/etc/sudoers` Deafults. + +**Note:** This information is crucial for a seamless experience when using Skipper with Docker on Linux. + + ## Usage Skipper can serve as your primary tool for your daily development tasks: + * Use `skipper build` to build the images defined by the Dockerfiles in your repository. All the images will be automatically tagged with the *COMMIT_ID*. * Use `skipper push` to publish your images. * Use `skipper images` to list your images. @@ -43,7 +64,8 @@ Skipper can serve as your primary tool for your daily development tasks: * Use `skipper shell` to get an interactive shell inside a container. ### Global Options -```bash + +```shell -v, --verbose Increase verbosity --registry URL of the docker registry --build-container-image Image to use as build container @@ -56,8 +78,10 @@ Skipper can serve as your primary tool for your daily development tasks: ``` ### [Build context explained](https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/) + Skipper allows you to add additional build contexts when running the build command, give them a name, and then access them inside a Dockerfile. The build context can be one of the following: + * Local directory – e.g. `--build-context project2=../path/to/project2/src` * Git repository – e.g. `--build-context qemu-src=https://github.com/qemu/qemu.git` * HTTP URL to a tarball – e.g. `--build-context src=https://example.org/releases/src.tar` @@ -72,7 +96,9 @@ RUN --mount=from=[name] … ``` ### Build + As a convention, skipper infers the docker images from the Dockerfiles in the top directory of your repository. For example, assuming that there are 3 Dockerfile in the top directory of the repository: + ``` Dockerfile.service1 Dockerfile.service2 @@ -80,92 +106,115 @@ Dockerfile.development ``` To build the image that corresponeds to `Dockerfile.service1`, run: -```bash + +```shell skipper build service1 ``` In the same way you can build the image corresponded to `Dockerfile.development`: -```bash + +```shell skipper build development ``` You can also build mutliple images with single command: -```bash + +```shell skipper build development service2 ``` A context path can be added to the build command, The build’s context is the files at a specified location PATH, the default is current directory: -```bash + +```shell skipper buid service1 --container-context /path/to/context/dir ``` If no image is specifed skipper will build all detected images: -```bash + +```shell skipper build ``` If you don't want to store all the Dockerfiles under the top directory of the project, you can specify the project's containers in skipper's config file (see below). ### Push + Once you've built the images of your repositories as described above. You can publish them by pushing them to the registry. To push the `service1` image, run: -```bash + +```shell skipper --registry some-registry push service1 ``` + Note that the registry in this command must be the same registry used while building the image. ### Images + To list local images of your repository, run: -```bash + +```shell skipper images ``` In order to also list also images that were pushed to the registry, run: -```bash + +```shell skipper --registry some-registry images -r ``` ### Rmi + To delete an image of your repository, run: -```bash + +```shell skipper rmi service1 ``` In order to delete the image from the registry, run: -```bash + +```shell skipper --registry some-registry rmi -r service1 ``` ### Make + You can execute a Makefile target inside a container. This is good for keeping the development in an isolated environment, without installing development tools on the host. Once a development container is defined and built, it can be shared among the team member, assuring all of them use exactly thg same development environment. Assuming your project has a Makefile with a `tests` target, you can run: -```bash + +```shell skipper --registry some-registry --build-container-image development --build-container-tag latest \ make tests ``` If your Makefile is not standard (i.e. `Makefile.arm32`) you can pass it to the make command: -```bash + +```shell skipper --registry some-registry --build-container-image development --build-container-tag latest \ make -f Makefile.arm32 tests ``` ### Run -You can also run arbitrary commands inside your containers. -```bash + +You can also run arbitrary commands inside your containers. + +```shell skipper --registry some-registry --build-container-image development --build-container-tag latest \ run gcc myprog.c -o myprog ``` ### Shell -You can get a shell inside your containers. -```bash + +You can get a shell inside your containers. + +```shell skipper --registry some-registry --build-container-image development --build-container-tag latest \ shell ``` ## Configuration File + Skipper allows you to define commonly used parameters in a configuration file `skipper.yaml` at the top directory of your repositry. + ```yaml registry: some-registry build-container-image: development @@ -201,41 +250,45 @@ build-container-tag: 'git:revision' ``` Using the above configuration file, we now can run a simplified version of the make command described above: -```bash + +```shell skipper make tests ``` ### Published ports + For `shell`, `run` & `make` commands: By default, when you run skipper on a linux machine it will use the host network and no mapping required. -For macos and windows machines where the host network is unsupported or for a custom network, you can publish a port and make it available to services outside of the container using the --publish or -p flag. +For macos and windows machines where the host network is unsupported or for a custom network, you can publish a port and make it available to services outside of the container using the --publish or -p flag. -```` +````shell skipper make -p 123:123 tests skipper make -p 123-130:123-130 tests ```` +### Environment variables -### Environment variables: For `shell`, `run` & `make` commands: You can use `-e` in order to pass environment variables to the container. -```` + +````shell skipper make -e regex=test1 tests ```` Your configuration file can contain environment variables, Skipper will set the specified environment variables in the container. -```` + +````yaml env: VAR: value ```` You can add an environment variables file (or multiple files) using `--env-file`. -This file should use the syntax =value (which sets the variable to the given value) or +This file should use the syntax =value (which sets the variable to the given value) or (which takes the value from the local environment), and # for comments. The variables defined in this file will be exported to the container. Such file can look like this: -```` +````shell $ cat env_file.env # This is a comment KEY1=value1 @@ -244,21 +297,24 @@ KEY3 ```` Skipper configuration file can include the environment variables file: -```` + +````yaml env_file: - /path/to/env_file1.env - /path/to/env_file2.env ```` +### Variable substitution -### Variable substitution: Skipper uses the variable values from the shell environment in which skipper is run. It’s possible to use environment variables in your shell to populate values For example, suppose the shell contains EXTERNAL_PORT=5000 and you supply this configuration: -```` + +````yaml env: EXTERNAL_PORT: $EXTERNAL_PORT ```` + When you run Skipper command with this configuration, Skipper looks for the EXTERNAL_PORT environment variable in the shell and substitutes its value in.In this example, Skipper resolves the `$EXTERNAL_PORT` to "5000" and will set EXTERNAL_PORT=5000 environment in the container. If an environment variable is not set, Skipper substitutes with an empty string. @@ -266,12 +322,14 @@ If an environment variable is not set, Skipper substitutes with an empty string. Both `$VARIABLE` and `${VARIABLE}` syntax are supported. Extended shell-style features, such as `${VARIABLE-default}` and `${VARIABLE/foo/bar}`, are not supported. You can use a `$$` (double-dollar sign) when your configuration needs a literal dollar sign. This also prevents Skipper from interpolating a value, so a `$$` allows you to refer to environment variables that you don’t want processed by Skipper. -```` + +````yaml env: VAR: $$VAR_NOT_INTERPOLATED ```` ### Shell Interpolation + Skipper supports evaluating shell commands inside its configuration file using `$(command)` notation. e.g. @@ -282,32 +340,38 @@ volumes: - $(which myprogram):/myprogram ``` +### Volumes -### Volumes: Skipper can bind-mount a host directory into the container. you can add volumes in the configuration file: -```` + +````yaml volumes: - /tmp:/tmp:rw - ${HOME}/.netrc:/root/.netrc - ${HOME}/.gocache:/tmp/.gocache ```` -### Workdir: +### Workdir + Skipper default to the the project directory as the working directory for the `run`, `make` and `shell` commands, you can override the workdir by specifying it in the configuration file: -```` + +````yaml workdir: /path/to/workdir ```` -### Workspace: +### Workspace + Skipper default to the the project base directory (e.g. /path/to/project/../) as the workspace for the `run`, `make` and `shell` commands, Note that the workspace directory is mounted by default. you can override the workspace directory by specifying it in the configuration file -```` + +````yaml workdir: $PWD ```` ### Skipper environment variables + Skipper sets environemnt variables to inform the user about the underline system: CONTAINER_RUNTIME_COMMAND - The container conmmand used to run the skipper container. podman/docker diff --git a/setup.cfg b/setup.cfg index 4bdc832..e4724f2 100644 --- a/setup.cfg +++ b/setup.cfg @@ -10,7 +10,13 @@ long_description_content_type = text/markdown requires_dist = setuptools requires_python = >=3 - +classifier = + Environment :: Console + Intended Audience :: Developers + Intended Audience :: Information Technology + License :: OSI Approved :: Apache Software License + Operating System :: POSIX :: Linux :: MacOS :: MacOS X + Programming Language :: Python :: 3 [files] packages = diff --git a/skipper/data/skipper-entrypoint.sh b/skipper/data/skipper-entrypoint.sh index 60cdb02..ee7e823 100755 --- a/skipper/data/skipper-entrypoint.sh +++ b/skipper/data/skipper-entrypoint.sh @@ -3,7 +3,6 @@ if ! [ -z "${SKIPPER_DOCKER_GID}" ];then HOME_DIR=${HOME} - SKIP_HOME_DIR_PARAM="" # if home directory already exists, useradd should not try to create it @@ -29,8 +28,13 @@ if ! [ -z "${SKIPPER_DOCKER_GID}" ];then usermod -G root ${SKIPPER_USERNAME} fi - - su -m ${SKIPPER_USERNAME} -c "$@" + if ! which sudo > /dev/null; then + su -m ${SKIPPER_USERNAME} -c "$@" + else + # for debian dsitros (maybe for others too) -m flag resets the PATH variable + # so we need to use sudo -E to preserve the PATH + sudo -sE -u ${SKIPPER_USERNAME} "$@" + fi else bash -c "$@" fi diff --git a/skipper/runner.py b/skipper/runner.py index ebde2c0..65b2331 100644 --- a/skipper/runner.py +++ b/skipper/runner.py @@ -138,22 +138,22 @@ def handle_networking(cmd, publish, net): def handle_volumes_bind_mount(docker_cmd, homedir, volumes, workspace): volumes = volumes or [] volumes.extend([f'{homedir}/.netrc:{homedir}/.netrc:ro', - f'{homedir}/.gitconfig:{homedir}/.gitconfig:ro', - f'{homedir}/.docker/config.json:{homedir}/.docker/config.json:ro']) + f'{homedir}/.gitconfig:{homedir}/.gitconfig:ro']) + + # required for docker buildkit and credentials + _add_path_if_exists(f'{homedir}/.docker', f'{homedir}/.docker', 'rw', volumes) # required for docker login (certificates) - if os.path.exists('/etc/docker'): - volumes.append('/etc/docker:/etc/docker:ro') + _add_path_if_exists('/etc/docker', '/etc/docker', 'ro', volumes) if utils.get_runtime_command() == utils.PODMAN: volumes.extend([ f'{workspace}:{workspace}:rw', f'{utils.get_extra_file("skipper-entrypoint.sh")}:/opt/skipper/skipper-entrypoint.sh:rw', ]) - if os.path.exists('/var/run/docker.sock'): - volumes.append('/var/run/docker.sock:/var/run/docker.sock:rw') - if os.path.exists('/var/lib/osmosis'): - volumes.append('/var/lib/osmosis:/var/lib/osmosis:rw') + + _add_path_if_exists('/var/run/docker.sock', '/var/run/docker.sock', 'rw', volumes) + _add_path_if_exists('/var/lib/osmosis', '/var/lib/osmosis', 'rw', volumes) else: volumes.extend([ f'{workspace}:{workspace}:rw', @@ -161,8 +161,7 @@ def handle_volumes_bind_mount(docker_cmd, homedir, volumes, workspace): f'{utils.get_extra_file("skipper-entrypoint.sh")}:/opt/skipper/skipper-entrypoint.sh', ]) # Will fail on Mac - if os.path.exists('/var/lib/osmosis'): - volumes.append('/var/lib/osmosis:/var/lib/osmosis:rw') + _add_path_if_exists('/var/lib/osmosis', '/var/lib/osmosis', 'rw', volumes) for volume in volumes: if ":" not in volume: @@ -214,6 +213,11 @@ def create_vol_localpath_if_needed(host_path): return True +def _add_path_if_exists(source, target, permissions, volumes): + if os.path.exists(source): + volumes.append(f'{source}:{target}:{permissions}') + + @contextmanager def _network(net): if utils.get_runtime_command() != "docker":