File not found by docker(-compose) when setting up dev environment

[jbothma]

Delena said she had trouble setting up - I'll let her fill in details if needed

[jbothma]

I think what's happening is the setup instructions don't wait for solr and postgres to be ready (which the normal running command does), but try to connect to them.

The first command in setup instructions after cloning is setting up the DB. This is what it looks like for me after cleaning up my setup (incl deleting volumes)

docker-compose run --rm ckan paster --plugin=ckan db init -c /ckan.ini
Creating volume "ckan-docker_solr-data" with default driver
Creating volume "ckan-docker_ckan-filestore" with default driver
Creating volume "ckan-docker_db-data" with default driver
Creating ckan-docker_solr_1            ... done
Creating ckan-docker_db_1              ... done
Creating ckan-docker_ckan-datapusher_1 ... done
Creating ckan-docker_redis_1           ... done
2019-05-23 07:34:57,980 ERROR [pysolr] Failed to connect to server at 'http://solr:8983/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json', are you sure that URL is correct? Checking it in a browser might help: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc174c9ebd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 366, in _send_request
    timeout=self.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 537, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 524, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 637, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc174c9ebd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2019-05-23 07:34:57,983 ERROR [ckan.lib.search.common] Failed to connect to server at 'http://solr:8983/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json', are you sure that URL is correct? Checking it in a browser might help: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc174c9ebd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
  File "/src/ckan/ckan/lib/search/common.py", line 57, in is_available
    conn.search(q="*:*", rows=1)
  File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 720, in search
    response = self._select(params, handler=search_handler)
  File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 418, in _select
    return self._send_request('get', path)
  File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 375, in _send_request
    raise SolrError(error_message % params)
SolrError: Failed to connect to server at 'http://solr:8983/solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json', are you sure that URL is correct? Checking it in a browser might help: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/ckan/select/?q=%2A%3A%2A&rows=1&wt=json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc174c9ebd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2019-05-23 07:34:57,984 WARNI [ckan.lib.search] Problems were found while connecting to the SOLR server
Traceback (most recent call last):
  File "/usr/local/bin/paster", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/src/ckan/ckan/lib/cli.py", line 333, in command
    self._load_config(cmd!='upgrade')
  File "/src/ckan/ckan/lib/cli.py", line 306, in _load_config
    self.site_user = load_config(self.options.config, load_site_user)
  File "/src/ckan/ckan/lib/cli.py", line 221, in load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/src/ckan/ckan/config/environment.py", line 99, in load_environment
    p.load_all()
  File "/src/ckan/ckan/plugins/core.py", line 129, in load_all
    unload_all()
  File "/src/ckan/ckan/plugins/core.py", line 182, in unload_all
    unload(*reversed(_PLUGINS))
  File "/src/ckan/ckan/plugins/core.py", line 210, in unload
    plugins_update()
  File "/src/ckan/ckan/plugins/core.py", line 121, in plugins_update
    environment.update_config()
  File "/src/ckan/ckan/config/environment.py", line 268, in update_config
    model.init_model(engine)
  File "/src/ckan/ckan/model/__init__.py", line 156, in init_model
    version_table = Table('migrate_version', meta.metadata, autoload=True)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 352, in __new__
    table._init(name, metadata, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 425, in _init
    self._autoload(metadata, autoload_with, include_columns)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 448, in _autoload
    self, include_columns, exclude_columns
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1670, in run_callable
    with self.contextual_connect() as conn:
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1737, in contextual_connect
    self.pool.connect(),
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 332, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 630, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 433, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 949, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 278, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 404, in __init__
    self.connection = self.__connect()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 530, in __connect
    connection = self.__pool._creator()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 95, in connect
    connection_invalidated=invalidated
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 89, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 376, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
sqlalchemy.exc.OperationalError: (OperationalError) could not connect to server: Connection refused
	Is the server running on host "db" (172.21.0.2) and accepting
	TCP/IP connections on port 5432?
 None None

After running that, I checked what actually got up and running:

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                    NAMES
fc22087de6ed        ckan-docker_solr              "docker-entrypoint.s…"   47 seconds ago      Up 44 seconds       0.0.0.0:8983->8983/tcp   ckan-docker_solr_1
dd6cca1940c0        redis:latest                  "docker-entrypoint.s…"   47 seconds ago      Up 44 seconds       6379/tcp                 ckan-docker_redis_1
513e1a2ceb77        ckan-docker_ckan-datapusher   "python /src/datapus…"   47 seconds ago      Up 44 seconds       0.0.0.0:8800->8800/tcp   ckan-docker_ckan-datapusher_1
e00a72353999        postgres:9.4                  "docker-entrypoint.s…"   47 seconds ago      Up 44 seconds       0.0.0.0:5433->5432/tcp   ckan-docker_db_1

Since solr and postgres as running, I tried the DB setup command again:

docker-compose run --rm ckan paster --plugin=ckan db init -c /ckan.ini
Starting ckan-docker_solr_1            ... 
Starting ckan-docker_ckan-datapusher_1 ... done
Starting ckan-docker_solr_1            ... done
2019-05-23 07:35:55,456 DEBUG [ckanext.extractor.model] Defining resource metadata table
2019-05-23 07:35:55,460 DEBUG [ckanext.extractor.model] Defining resource metadatum table
Initialising DB: SUCCESS

[delenamalan]

What I get on a new docker-compose up

$ docker-compose up                                                                                                          [9:54:42]
Creating network "treasury-ckan_default" with the default driver
Creating treasury-ckan_ckan-datapusher_1 ... done
Creating treasury-ckan_db_1              ... done
Creating treasury-ckan_redis_1           ... done
Creating treasury-ckan_solr_1            ... done
Creating treasury-ckan_ckan-worker_1     ... 
Creating treasury-ckan_ckan_1            ... error

ERROR: for treasury-ckan_ckan_1  Cannot start service ckan: OCI runtime create failed: 
container_linux.go:345: starting container process caused "process_linux.go:424: container init caused 
\"rootfs_linux.go:58: mounting \\\"/home/foiuglkj/Workspace/openup/treasury-ckan/ckan.ini\\\" to rootfs \\\"/mnt
/sda1/var/lib/docker/overlay2/9fd51096f4b9d0bb26134f838e9e24527b3a73c49525a6dd16e10a932c71d6a4
/merged\\\" at \\\"/mnt/sda1/var/lib/docker/overlay2
/9fd51096f4b9d0bb26134f838e9e24527b3a73c49525a6dd16e10a932c71d6a4/merged/ckan.ini\\\" 
causedCreating treasury-ckan_ckan-worker_1     ... error
 exists and is the expected type

ERROR: for treasury-ckan_ckan-worker_1  Cannot start service ckan-worker: OCI runtime create failed: 
container_linux.go:345: starting container process caused "process_linux.go:424: container init caused 
\"rootfs_linux.go:58: mounting \\\"/home/foiuglkj/Workspace/openup/treasury-ckan/ckan.ini\\\" to rootfs \\\"/mnt
/sda1/var/lib/docker/overlay2/8f79ec531babbd490a82504d3346b41643207e00a6d95af9c4aa2e13fc2afaec
/merged\\\" at \\\"/mnt/sda1/var/lib/docker/overlay2
/8f79ec531babbd490a82504d3346b41643207e00a6d95af9c4aa2e13fc2afaec/merged/ckan.ini\\\" caused 
\\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the 
specified host path exists and is the expected type


ERROR: for ckan  Cannot start service ckan: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/home/foiuglkj/Workspace/openup/treasury-ckan/ckan.ini\\\" to rootfs \\\"/mnt/sda1/var/lib/docker/overlay2/9fd51096f4b9d0bb26134f838e9e24527b3a73c49525a6dd16e10a932c71d6a4/merged\\\" at \\\"/mnt/sda1/var/lib/docker/overlay2/9fd51096f4b9d0bb26134f838e9e24527b3a73c49525a6dd16e10a932c71d6a4/merged/ckan.ini\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ERROR: for ckan-worker  Cannot start service ckan-worker: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/home/foiuglkj/Workspace/openup/treasury-ckan/ckan.ini\\\" to rootfs \\\"/mnt/sda1/var/lib/docker/overlay2/8f79ec531babbd490a82504d3346b41643207e00a6d95af9c4aa2e13fc2afaec/merged\\\" at \\\"/mnt/sda1/var/lib/docker/overlay2/8f79ec531babbd490a82504d3346b41643207e00a6d95af9c4aa2e13fc2afaec/merged/ckan.ini\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

I can confirm all oft those files exist on my host machine.

[jbothma]

docker version
Client:
 Version:           18.09.6-ce
 API version:       1.39
 Go version:        go1.12.4
 Git commit:        481bc77156
 Built:             Sat May 11 06:11:03 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.6-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.12.4
  Git commit:       481bc77156
  Built:            Sat May 11 06:10:35 2019
  OS/Arch:          linux/amd64
  Experimental:     false

docker-compose version
docker-compose version 1.24.0, build unknown
docker-py version: 3.7.2
CPython version: 3.7.3
OpenSSL version: OpenSSL 1.1.1b  26 Feb 2019

[jbothma]

I managed to get this working by following the setup instructions in the readme inside Docker-in-Docker

I don't think that's a sensible way to have to use this (probably) but it suggests that it works at least on alpine linux (dind base image) and arch linux and some ubuntu (that jaron was using) so we need to figure out what it is about Delena's environment that makes it not work.

docker exec -ti dind /bin/ash

dind-treasury-ckan.txt