"Page not found" error when using with balena

[mlxyz]

Describe the bug
I am trying to use the container in a balena cloud environment.
This is the excerpt from the docker compose file:

  backup:
    image: offen/docker-volume-backup:latest
    restart: always
    volumes:
      - paperless_media:/backup/paperless-media:ro
      - paperless_data:/backup/paperless-data:ro
      - paperless_pgdata:/backup/paperless-pgdata:ro
      - caddy_data:/backup/caddy-data:ro
      - caddy_config:/backup/caddy-config:ro
    privileged: true
    labels:
      io.balena.features.balena-socket: "1"
    networks:
      - paperless
      - caddy

When running the backup, I get the following error message:
time=2024-02-24T10:49:55.700Z level=ERROR msg="Fatal error running command: page not found" error="main.(*command).runAsCommand: error running script: main.runScript.func4: error running script: main.(*script).stopContainersAndServices: error querying for services: Error response from daemon: page not found"

DOCKER_HOST is set:

~ # printenv | grep DOCKER
DOCKER_HOST=unix:///host/run/balena-engine.sock

To Reproduce
Steps to reproduce the behavior:

1. Setup balena cloud device with backup container and consuming container
2. Start backup

Expected behavior
Backup running without any issues.

Version (please complete the following information):

• Image Version: v2.38.1
• Docker Version: v20.10.43
• Docker Compose Version (if applicable):

Additional context
Balena-engine:

Client:
 Version:           v20.10.43
 API version:       1.41
 Go version:        go1.17.13
 Git commit:        0be8046a1c191ad789a65d7d8429d899d6bc9fbe
 Built:             Tue Feb  6 08:42:34 2024
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          v20.10.43
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.13
  Git commit:       0be8046a1c191ad789a65d7d8429d899d6bc9fbe
  Built:            Tue Feb  6 08:42:34 2024
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.6+unknown
  GitCommit:        
 runc:
  Version:          unknown
  GitCommit:        
 balena-engine-init:
  Version:          0.13.0
  GitCommit:        949e6fa-dirty

[m90]

I don't know anything about Balena, its compatibility with Docker and what concepts it comes with.

Judging from what's going on in the code, there could be two things happening:

• Balena reports it's running in Swarm mode when it's not, making this tool call the services endpoint which does not exist when not running in Swarm mode
• Balena is running in Swarm mode, but does not expose a services endpoint, thus you get a 404

To further debug this, could you provide the output of

• docker info --format="{{ .Swarm | json }}"
• docker service ls

[mlxyz]

Thanks for digging into this.
I can only provide outputs for the corresponding balena-engine commands, as the docker cli is not available on balena-managed devices.
Output of balena-engine info --format="{{ .Swarm | json }}":
{"NodeID":"","NodeAddr":"","LocalNodeState":"","ControlAvailable":false,"Error":"","RemoteManagers":null}

balena-engine service ls is not available. These commands are:

root@central:~# balena-engine --help

Usage:  balena-engine [OPTIONS] COMMAND

A self-sufficient runtime for containers

Options:
      --config string      Location of client config files (default "/home/root/.balena-engine")
  -c, --context string     Name of the context to use to connect to the daemon (overrides DOCKER_HOST env var and default context
                           set with "docker context use")
  -D, --debug              Enable debug mode
  -H, --host list          Daemon socket(s) to connect to
  -l, --log-level string   Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
      --tls                Use TLS; implied by --tlsverify
      --tlscacert string   Trust certs signed only by this CA (default "/home/root/.balena-engine/ca.pem")
      --tlscert string     Path to TLS certificate file (default "/home/root/.balena-engine/cert.pem")
      --tlskey string      Path to TLS key file (default "/home/root/.balena-engine/key.pem")
      --tlsverify          Use TLS and verify the remote
  -v, --version            Print version information and quit

Management Commands:
  builder     Manage builds
  container   Manage containers
  image       Manage images
  manifest    Manage Docker image manifests and manifest lists
  network     Manage networks
  system      Manage Docker
  trust       Manage trust on Docker images
  volume      Manage volumes

Commands:
  attach      Attach local standard input, output, and error streams to a running container
  build       Build an image from a Dockerfile
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  events      Get real time events from the server
  exec        Run a command in a running container
  export      Export a container's filesystem as a tar archive
  history     Show the history of an image
  images      List images
  import      Import the contents from a tarball to create a filesystem image
  info        Display system-wide information
  inspect     Return low-level information on Docker objects
  kill        Kill one or more running containers
  load        Load an image from a tar archive or STDIN
  login       Log in to a Docker registry
  logout      Log out from a Docker registry
  logs        Fetch the logs of a container
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  ps          List containers
  pull        Pull an image or a repository from a registry
  push        Push an image or a repository to a registry
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  rmi         Remove one or more images
  run         Run a command in a new container
  save        Save one or more images to a tar archive (streamed to STDOUT by default)
  search      Search the Docker Hub for images
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  version     Show the Docker version information
  wait        Block until one or more containers stop, then print their exit codes

Run 'balena-engine COMMAND --help' for more information on a command.

To get more help with docker, check out our guides at https://docs.docker.com/go/guides/

[m90]

Thanks for providing the output, seems the first scenario applies and this check

docker-volume-backup/cmd/backup/stop_restart.go

Line 96 in f4c6112

isDockerSwarm := dockerInfo.Swarm.LocalNodeState != "inactive"

now fails. This means, the tool thinks your engine is running in Swarm mode, and tries to list services, but obviously fails to do so.

While this would be easy to fix by just changing the condition to check for an empty string or even flip the conditions to check against valid swarm states, I am a bit worried about this could become a rabbit hole of sorts, chasing down tiny incompatibilities like this one between Docker and Balena. If I understand the Docker OpenAPI spec correctly, SwarmInfo should be populated always (which Docker Engine also adheres to), but apparently Balena doesn't do it, which leaves me worrying this will also happen elsewhere. Do you have any kind of document about Docker / Balena compatibility "guarantees" or something that I could check in order to get an idea about how big the differences are in areas that this tool is interfacing with? Do you know if there is a way to run "Balena in Docker" so it could be used to run tests against?

[m90]

All I could find is:

• https://engine-docs.balena.io/#transitioning-from-docker-ce which is only semi helpful (it should work ™️)
• https://github.com/balena-io-experimental/balena-engine-docker which looks like it could be used for testing (unfortunately it's not licensed: What's the license for the code in this repo? balena-io-experimental/balena-engine-docker#2)

[mlxyz]

@m90 Yes, afaik there is no comprehensive documentation about compatibility between balena engine and docker engine. In this thread, it is noted that balena engine is a moby-compatible container engine.

Your PR looks promising though - brilliant! 🎉

[m90]

In case your setup allows you to do so, you can try checking out the code from this branch #376 and build an image off that. I do think "it should work ™️"

I'm still struggling to get a non-nausea-inducing test setup working (running Balena-in-Docker in Docker-in-Docker and such 💫). Also having to build the Balena-in-Docker image on each CI run currently adds a whopping extra of 22 minutes for building the image, which is kind of a show stopper. I.e. this might take a little while still, but maybe you can get it working with that self-built image already.

[mlxyz]

Just tried to build and run it, got a exec /usr/bin/backup: exec format error message - I probably built it for the wrong arch. I'll try again tomorrow!

[mlxyz]

@m90 I can confirm everything is working with the balena-engine branch! 🎉

[m90]

This is now fixed in v2.38.2