Dagster suddenly cannot communicate with Docker to launch run containers

[clayheaton]

I've been running self-hosted Dagster with Docker Compose and 5 code locations for over a year without any problems, updating regularly. This is on Ubuntu 22.04.

Today, out of the blue, Dagster no longer can communicate with Docker to launch containers. This is the error:

docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker

This is how I have my run launcher configured. I have not touched this in over a year:

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    network: dagster_docker_network
    container_kwargs:
      volumes: # Make docker client accessible to any launched containers as well
        - /var/run/docker.sock:/var/run/docker.sock

If I just ssh to the VM where Dagster is running, I can do this in a python repl:

import docker
client = docker.from_env()

However, if I docker exec -it dagster_daemon bash into the dagster_daemon container, trying the same code leads to the error shown above. The dagster_daemon is defined like this in my docker compose file:

  dagster_daemon:
    env_file:
      - ${PWD}/.env
    build:
      context: .
      dockerfile: ./Dockerfile_dagster
    entrypoint:
      - dagster-daemon
      - run
    container_name: dagster_daemon
    profiles: ["nondata", "all"]
    restart: on-failure
    environment:
      DAGSTER_POSTGRES_URL: ${DAGSTER_POSTGRES_CONN_STRING}
    volumes: # Make docker client accessible so we can launch containers using host docker
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/io_manager_storage:/tmp/io_manager_storage
    networks:
      - dagster_docker_network
    depends_on:
      - code_loc1
      - code_loc2

Any ideas? I have no idea why this suddenly stopped working. I've rebooted the VM, upgraded docker, etc... but none of it is enabling the dagster run launcher to communicate with docker...

[gibsondan]

Hi clay - it looks like you are hitting this upstream issue with the docker python client: docker/docker-py#3256

You can resolve it by pinning the version of requests that you're using to <2.32.0 (until docker lands their hotfix that they are working on)

[clayheaton]

This fixed it. Thank you so much! :)

[joaofonsecapromptly]

Hey guys, despite my requests package being at 2.31.0, it still has the same issue.
What version of docker and dagster-docker are you using?

[pitaleabhishek]

same here. everything was fine until last week. These are my libraries:
[tool.poetry]
name = "dagster-dbt-snowflake-poc"
version = "0.1.0"
description = ""
authors = ["None"]
readme = "README.md"
package-mode = false

[tool.poetry.dependencies]
python = "3.10.14"
agate = "1.7.1"
alembic = "1.13.1"
annotated-types = "0.6.0"
asn1crypto = "1.5.1"
attrs = "23.2.0"
babel = "2.14.0"
certifi = "2024.2.2"
cffi = "1.16.0"
charset-normalizer = "3.3.2"
click = "8.1.7"
colorama = "0.4.6"
coloredlogs = "14.0"
croniter = "2.0.3"
cryptography = "42.0.5"
dagster = "1.7.0"
docker = "7.0.0"
dagster-dbt = "0.23.0"
dagster-docker = "0.23.0"
dbt-core = "1.7.11"
dbt-extractor = "0.5.1"
dbt-semantic-interfaces = "0.4.4"
dbt-snowflake = "1.7.3"
docstring-parser = "0.16"
filelock = "3.13.3"
fsspec = "2024.3.1"
greenlet = "3.0.3"
grpcio = "1.62.1"
grpcio-health-checking = "1.62.1"
humanfriendly = "10.0"
idna = "3.6"
importlib-metadata = "6.11.0"
isodate = "0.6.1"
jaraco-classes = "3.4.0"
jeepney = "0.8.0"
jinja2 = "3.1.3"
jsonschema = "4.21.1"
jsonschema-specifications = "2023.12.1"
keyring = "24.3.1"
leather = "0.4.0"
logbook = "1.5.3"
mako = "1.3.2"
markdown-it-py = "3.0.0"
markupsafe = "2.1.5"
mashumaro = "3.12"
mdurl = "0.1.2"
minimal-snowplow-tracker = "0.0.2"
more-itertools = "10.2.0"
msgpack = "1.0.8"
networkx = "3.3"
orjson = "3.10.0"
packaging = "24.0"
parsedatetime = "2.6"
pathspec = "0.11.2"
pendulum = "3.0.0"
platformdirs = "3.11.0"
protobuf = "4.25.3"
pycparser = "2.22"
pydantic = "2.6.4"
pydantic-core = "2.16.3"
pygments = "2.17.2"
pyjwt = "2.8.0"
pyopenssl = "24.1.0"
python-dateutil = "2.9.0.post0"
python-dotenv = "1.0.1"
python-slugify = "8.0.4"
pytimeparse = "1.1.8"
pytz = "2024.1"
pyyaml = "6.0.1"
referencing = "0.34.0"
requests = "2.31.0"
rich = "13.7.1"
rpds-py = "0.18.0"
secretstorage = "3.3.3"
shellingham = "1.5.4"
six = "1.16.0"
snowflake-connector-python = "3.7.1"
sortedcontainers = "2.4.0"
sqlalchemy = "2.0.29"
sqlglot = "23.8.1"
sqlglotrs = "0.2.0"
sqlparse = "0.4.4"
structlog = "24.1.0"
tabulate = "0.9.0"
text-unidecode = "1.3"
time-machine = "2.14.1"
tomli = "2.0.1"
tomlkit = "0.12.4"
toposort = "1.10"
tqdm = "4.66.2"
typer = "0.12.3"
typing-extensions = "4.11.0"
tzdata = "2024.1"
universal-pathlib = "0.2.2"
urllib3 = "1.26.18"
watchdog = "4.0.0"
zipp = "3.18.1"
dagster-postgres = "0.23.0"
dagster-shell = "0.23.0"
pandas = "^2.2.2"
dagster-webserver = "1.7.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[clayheaton]

I'm using dagster==1.7.5 and the associated libraries released with that version. Otherwise, I have dagster-docker>=0.23.5, docker>=6.1.3, and requests<=2.31.0.

If there's a chance that docker is using an intermediate container with the wrong version, maybe try docker system prune -a (please check you're not causing yourself a problem before doing this) and force a complete rebuild.