Loki docker driver can not push to server. Gives 404 errors.

[pgrond]

I've configured the docker daemon to send all docker logs to my Loki server. I've also a promtail container running, wit the same endpoint, and those logs get pushed to loki without a problem.
The docker logs are not received though.

To Reproduce

1. Install Loki docker driver and check status
   ID NAME DESCRIPTION ENABLED
   1949660d95d7 loki:latest Loki Logging Driver true

2. Configure docker daemon to use Loki driver

{
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://loki.mydomain.nl:3100/loki/api/v1/push",
        "loki-batch-size": "400"
    }
}

3. Restart docker daemon
   systemctl restart docker

Expected behavior
See the docker logs in Grafana. But no logs are collected.
I can see in the syslog messages like:

ubuntu-8gb-nbg1-1 dockerd[29464]: time="2020-05-21T16:26:21+02:00" level=info msg="level=error ts=2020-05-21T14:26:21.051575851Z caller=client.go:247 container_id=5edb13014e9a770c00bc4510883e30ff9ff108fd76de26aec271da83302c8539 component=client host=loki.mydomain.nl msg="final error sending batch" status=404 error="server returned HTTP status 404 Not Found (404): 404 page not found"" plugin=1949660d95d7235dccf4c319f34d8178f2d6a706b741b37abca66548cca5e7ac

Promtail works just fine with this config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: https://loki.mydomain.nl:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  pipeline_stages:
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: journal
  pipeline_stages:
  journal:
    path: /var/log/journal
    labels:
      job: systemd-journal
  relabel_configs:
    - source_labels: ['__journal__systemd_unit']
      target_label: 'unit'

Am I missing something? As far as I can see I followed the documentation on https://github.com/grafana/loki/tree/v1.5.0/docs/clients/docker-driver

Environment:
All dockers run with docker compose files, no Kubernetes

[cyriltovena]

Can you add info logs and show us the line where this appears:

	level.Info(d.logger).Log("msg", "starting logging driver for container", "id", logCtx.ContainerID, "config", fmt.Sprintf("%+v", logCtx.Config), "file", file, "logpath", logCtx.LogPath)

Feel like there could be some dns issue with your driver ? Promtail and Docker driver actually use the same go code to send data. So it's weird one is working only.

[cyriltovena]

actually info logs are on by default :)

[pgrond]

May 21 13:08:07 ubuntu-8gb-nbg1-1 dockerd[23833]: time="2020-05-21T13:08:07+02:00" level=info msg="level=info ts=2020-05-21T11:08:07.083633374Z caller=driver.go:77 msg=\"starting logging driver for container\" id=5276d885a93704bf96a6f5ac70839ce45f7244cfb96def0bb1edce5bc957ce72 config=\"map[loki-batch-size:400 loki-url:https://loki.mydomain.nl/loki/api/v1/push]\" file=/run/docker/logging/b2f03577c63997d1249dd67214bb62feccec9f62df53dbc1f32e9b559557f6b9 logpath=/var/log/docker/5276d885a93704bf96a6f5ac70839ce45f7244cfb96def0bb1edce5bc957ce72/json.log" plugin=1949660d95d7235dccf4c319f34d8178f2d6a706b741b37abca66548cca5e7ac

[cyriltovena]

Well you might have seen it but the logs shows the port is missing, any idea why ?

[pgrond]

Ah, that explains probably why it is not working. But in daemon.json the port is there.... (see first post)
I don't know why or where that gets stripped.
Any hint where to look is appreciated ;)

Maybe some OS info might help:

cat /etc/lsb-release:
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

docker version:
Version: 19.03.7
API version: 1.40
Go version: go1.12.17
Git commit: 7141c199a2
Built: Wed Mar 4 01:22:36 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.7
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: 7141c199a2
Built: Wed Mar 4 01:21:08 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

[cyriltovena]

Not sure I've setup the deamon too and it works fine:

{
  "debug": true,
  "experimental": false,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://localhost:3100/loki/api/v1/push",
        "loki-batch-size": "400"
    }
}

2020-05-25T18:59:26Z dockerd time="2020-05-25T18:59:26Z" level=info msg="level=info ts=2020-05-25T18:59:26.496781638Z caller=driver.go:77 msg=\"starting logging driver for container\" id=438dece7d4e6e5dc93261fa80907be06dfdd6ec9d1955c7c3c1bfaa260a6e5c9 config=\"map[loki-batch-size:400 loki-url:https://localhost:3100/loki/api/v1/push]\" file=/run/docker/logging/8858f63a295909f9b1fb3fbd153b6c4586409fd24e3b7cc6dffdfac8c1e2f082 logpath=/var/log/docker/438dece7d4e6e5dc93261fa80907be06dfdd6ec9d1955c7c3c1bfaa260a6e5c9/json.log" plugin=aed2da1c60c42616dcc896a0721cd7dc47495fbcf22d47ec751e39d3f55b1d42

see the loki-url contains the port. I'm using 19.03.8.

Not sure what's up, but must be either your config or your dockerd.

[pgrond]

Ok, made my loki instance available on port 443... Fixes this for now. Don't know what the issue is, but this works for me now.