Promtail failed to run as a push receiver with metric errors

[cheesengoon]

Describe the bug
Encounter the following error when attempted to run Promtail v2.2.1 as a push receiver. The idea is to try receiving logs sent by another Promtail instance on another Linux server.

The following is the error log that I received.

./promtail-linux-amd64 -config.file ./promtail-config.yaml -log.level=debug

level=info ts=2021-06-01T04:26:50.681808163Z caller=pushtarget.go:78 msg="starting push server" job=promtail-push
panic: descriptor Desc{fqName: "promtail_promtail-push_request_duration_seconds", help: "Time (in seconds) spent serving HTTP requests.", constLabels: {}, variableLabels: [method route status_code ws]} is invalid: "promtail_promtail-push_request_duration_seconds" is not a valid metric name
 
goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0000b4820, 0xc0004d18a0, 0x1, 0x1)
        /src/loki/vendor/github.com/prometheus/client_golang/prometheus/registry.go:403 +0xad
github.com/prometheus/client_golang/prometheus.MustRegister(...)
        /src/loki/vendor/github.com/prometheus/client_golang/prometheus/registry.go:178
github.com/weaveworks/common/server.New(0xc000058fe0, 0x16, 0x0, 0x0, 0xdac, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /src/loki/vendor/github.com/weaveworks/common/server/server.go:204 +0x4e3
github.com/grafana/loki/pkg/promtail/targets/lokipush.(*PushTarget).run(0xc0005d1680, 0xc0001eec80, 0x2bad100)
        /src/loki/pkg/promtail/targets/lokipush/pushtarget.go:87 +0x278
github.com/grafana/loki/pkg/promtail/targets/lokipush.NewPushTarget(0x31b4540, 0xc000a82ea0, 0x31dcb60, 0xc0004d1790, 0x0, 0x0, 0x0, 0xc000524c10, 0xd, 0xc0001ee780, ...)
        /src/loki/pkg/promtail/targets/lokipush/pushtarget.go:69 +0x247
github.com/grafana/loki/pkg/promtail/targets/lokipush.NewPushTargetManager(0x31f87e0, 0xc0000b4820, 0x31b4540, 0xc000a82ea0, 0x7f6a1c6f7da0, 0xc000a8da00, 0xc000a9e3c0, 0x1, 0x1, 0x0, ...)
        /src/loki/pkg/promtail/targets/lokipush/pushtargetmanager.go:47 +0x374
github.com/grafana/loki/pkg/promtail/targets.NewTargetManagers(0x31b4c60, 0xc0005d0960, 0x31f87e0, 0xc0000b4820, 0x31b4540, 0xc000a82ea0, 0x2540be400, 0xc000053320, 0x21, 0xc000a80000, ...)
        /src/loki/pkg/promtail/targets/manager.go:174 +0x1095
github.com/grafana/loki/pkg/promtail.New(0x0, 0x0, 0x0, 0x0, 0x2378, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /src/loki/pkg/promtail/promtail.go:86 +0x585
main.main()
        /src/loki/cmd/promtail/main.go:106 +0x276

To Reproduce
Steps to reproduce the behavior:

1. Download Promtail v2.2.1 and unzip
2. Configure promtail-config.yaml as below

server:
   http_listen_port: 9080
   grpc_listen_port: 0
positions:
   filename: ./positions.yaml # This location needs to be writeable by promtail.
client:
   url: http://<loki ip address>:3100/loki/api/v1/push     
scrape_configs:
   - job_name: promtail-push
     loki_push_api:
       server:
         http_listen_port: 3500
         grpc_listen_port: 0
       labels:
         pushserver: promtail-server

3. Run the following command,

./promtail-linux-amd64 -config.file ./promtail-config.yaml -log.level=debug

Expected behavior
Startup successfully as a push receiver server listening on port 3500.

Environment:
Infrastructure: Linux RHEL 7

[dannykopping]

Thanks for reporting this issue @cheesengoon.

Here is the problem: https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/lokipush/pushtarget.go#L82
Your job name promtail-push is being prepended to the request_duration_seconds metric.

Promtail uses your job-name to setup a number of Prometheus metrics.
Please follow the Prometheus naming guidelines to solve this issue

tl;dr
Use an underscore instead of a hyphen in your job-name.