Can't scrape journal on Raspberry Pi

[hairyhenderson]

Describe the bug

On a Raspberry Pi running the official Raspbian bistro, promtail doesn't support reading from journal:

WARNING!!! Journal target was configured but support for reading the systemd journal is not compiled into this build of promtail!

It seems like this was done in #888 as a temporary measure, though it's not entirely clear why (is it because no non-AMD64 hardware is available for the CI builds?). Having to enable CGO when cross-compiling can be a pain, so maybe this is why?

It'd be super useful to be able to use the released armhf promtail binary to scrape logs on a Pi, though!

To Reproduce

1. run promtail with a journal configuration
2. see the logs and notice that it doesn't scrape the journal

Expected behavior

It should scrape the journal 😉

Environment:

• Infrastructure: Raspberry Pi

Screenshots, Promtail config, or terminal output

relevant config is:

scrape_configs:
- job_name: journald
  journal: {}

[hairyhenderson]

It's maybe worth noting as well that I've been able to get the build to start by removing the lines added in #888. However I'm running into memory constraints on my Raspberry Pi 3B (4 cores, 1GB RAM). After getting by some OOMs by building dependent packages directly, linking fails:

$ GOMAXPROCS=1 CGO_ENABLED=1 go build -ldflags "-s -w -X github.com/grafana/loki/pkg/build.Branch=HEAD -X github.com/grafana/loki/pkg/build.Version=v1.2.0-WIP -X github.com/grafana/loki/pkg/build.Revision=ccef3da2 -X github.com/grafana/loki/pkg/build.BuildUser=pi@mypi -X github.com/grafana/loki/pkg/build.BuildDate=2019-12-30T01:33:41Z" -tags netgo -mod=vendor -o cmd/promtail/promtail ./cmd/promtail
# github.com/grafana/loki/cmd/promtail
/usr/local/go/pkg/tool/linux_arm/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory

My gut feeling is that all the different SD dependencies are adding lots of bloat at compile time. I wonder if there's a way to disable them selectively at compile time... For my use-case I don't need the majority of the SD mechanisms.

[hairyhenderson]

Update: got it building after I added a whole pile of swap (with an external drive, to spare the poor SD card 😅)

[rfratto]

Thanks for reporting! We did have to disable it because we were using a cross-compiler for the ARM64 builds. and I forgot about it being disabled once we switched to using native ARM machines 🙂

[hairyhenderson]

Thanks @rfratto!

[ikogan]

I'm getting this error on 1.5.0 and the thread makes me think that 1.5.0 should be built with journal support...?

[Alpha200]

I have the same issue on 1.5.0 with the arm build.

[slim-bean]

How are you running promtail? Are you running from the binary or from the docker image?

[Alpha200]

I'm running it over the binary (arm, not arm64).

[ikogan]

Also running arm, not arm64.

[slim-bean]

the binaries are cross compiled with cgo disabled because it's a huge pain in the butt to cross compile with cgo, we would need a C cross compiler to make it work. A quick "fix" would be to use the docker image or even pull the promtail binary out of the docker image and see if that works.

The long term solution is going to be figuring out how to build arm binaries with cgo, not sure when we will be able to get to this.

[Alpha200]

I've tried to copy the binary out of the docker image, but now the binary fails to start with the following error message: fatal: systemstack called from unexpected goroutine.

I do not want to install docker on the Pis and sadly I'm stuck at Raspbian 9 because of some legacy software so I can not try it with podman.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[fuomag9]

This issue is still valid, I can't read journal on a raspberry pi

[willianpaixao]

Yeah, it's getting more relevant over time since there are more ARM based systems being released and used.

[blockloop]

I know this isn't ideal, but you can capture - what seems to be all - systemd logs
on raspberrypi from /var/log/daemon.log. I'm using the following config to capture
the unit from the log line.

  - job_name: system
    static_configs:
    - targets:
       - localhost
      labels:
       job: varlogs
       host: {{ ansible_hostname }}
       __path__: /var/log/*.log
    pipeline_stages: 
    - match:
        selector: '{filename="/var/log/daemon.log"}'
        stages:
        - regex:
            expression: "raspberrypi (?P<unit>[^[]+)"
        - labels:
            unit:

[milesstoetzner]

Another way is to start promtail as syslog server and receive journald logs using rsyslog.

See here for more information: https://grafana.com/docs/loki/latest/clients/promtail/configuration/#syslog

[rgl]

Any news about this landing on a Release? promtail 2.3.0 arm64 is not yet able to scrape Journal logs in Rasperry Pi.

[clothespin]

Coming here from #4380 which was closed as a dupe of this issue. I'm pretty sure the root cause is this bit of drone.jsonnet, which references this Dockerfile. This exists specifically to work around the issue where the journal libs are not available. The confusing part is, it doesn't seem to work in arm32 either, which means the above referenced fix doesn't fix the bug. Here was what I saw on running the latest promtail release for arm32:

pi@octopi:~ $ uname -a
Linux octopi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux
pi@octopi:~ $ cat promtail-test-cfg.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://REDACTED:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
pi@octopi:~ $ ./promtail-linux-arm -config.file ./promtail-test-cfg.yaml
level=warn ts=2021-10-12T02:49:33.554621147Z caller=journaltargetmanager.go:29 msg="WARNING!!! Journal target was configured but support for reading the systemd journal is not compiled into this build of promtail!"
level=info ts=2021-10-12T02:49:33.557969772Z caller=server.go:239 http=[::]:9080 grpc=[::]:36307 msg="server listening on addresses"
level=info ts=2021-10-12T02:49:33.558905808Z caller=main.go:119 msg="Starting Promtail" version="(version=, branch=, revision=)"

Not sure this makes sense as a kind/feature because I think this was meant to be working in the release build, at least for arm32. Shouldn't this be a bug rather than a feature request since the feature is supposed to be already shipped but is broken? Or am I misinterpreting what's already there? I could see the argument for arm64 support being a separate feature from arm32 but at least the latter is a bug.

[rfratto]

Shameless self-pitch: the Grafana Agent imports promtail and supports reading from journald on Raspberry Pis thanks to a different build process. I'm not a Loki maintainer anymore, but from what I remember of this issue, the main problem was that cross-compiled versions of Promtail (that aren't built in Docker) have CGO disabled. This isn't a problem with grafana/agent.

[kavirajk]

Thanks @rfratto!

So if we decide to do this, one of the easy option would be to use rfratto/seego build image to built those binaries!.

Putting it here for reference: https://github.com/grafana/agent/blob/main/Makefile#L244-L261

[saibottob]

Hello
I would like to ask, if this issue has been resolved or not?
Because I just downloaded the latest version of promtail for a arm build and I received this messages, that it cannot scrape journals.
Or exists there a workaround?

Best Regards

[damm]

The workaround is to build it yourself. If you have the systemd-dev package installed promtail will support reading from the systemd journal out of the box.

[woodz-]

A dumb question:
when the journal gets scraped (issue solved), where do I need to place my journal files to be picked up by promtail? I have promtail running in a container on a aarch64 pi, which is the loki sample with a tweaked promtail-local-config.yaml:

---
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://gateway:3100/loki/api/v1/push
    tenant_id: tenant1

scrape_configs:
  - job_name: journal 
    journal:
      max_age: 5000h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

The promtail container comes with no systemd support. I would think to put my journal files into the promtail container in /var/log/journal directory.

But systemd-journald needs to be there and started to initiate the journal. Is it right to assume the next thing to do is to get systemd running inside the promtail container?
Or is there another approach?

[bLd75]

Readme is quite outdated, this should help those who want to build themselves as a workaround:
CGO_ENABLED=1 go build -tags netgo --tags=promtail_journal_enabled -o clients/cmd/promtail/promtail ./clients/cmd/promtail

[damm]

You can also run the following:

cd clients/cmd/promtail
apt install libsystemd-dev -y
make BUILD_IN_CONTAINER=false promtail

It would be fantastic if the releases contained a promtail and could scrape the journal by default. I deploy loki using ansible so I have to replace the promtail binary after a run :(

[bLd75]

+1 I had to change my ansible to download from custom location as a workaround

[chrisguidry]

This is an atrocious hack, but this script can help you pull a compiled binary from the published Docker image, even from a machine that isn't ARM:

#!/bin/bash
TAG=main-2e5622b

docker pull --platform linux/arm grafana/promtail:$TAG
id=$(docker create --platform linux/arm grafana/promtail:$TAG)
docker cp $id:/usr/bin/promtail - > promtail.tar
docker rm $id
tar xvf promtail.tar
rm promtail.tar

If you run this, you'll be left with the ARM binary promtail in your current directory. I can confirm it includes journal support when run on a Raspberry Pi 3 (armhf).

Maintainers: Thank you for all your hard work! Having the normal release .deb include journal support would be very nice if possible.

[jeschkies]

@chrisguidry done 😊

[chrisguidry]

@chrisguidry done 😊

You guys rock, and this is some great software, btw!

[umglurf]

@jeschkies I downloaded the new 2.7.4 release binary for arm64 (promtail-linux-arm64.zip) and it still complains about journald support missing. Is there something else missing in the builds?

[jeschkies]

@umglurf that is interesting. It was reported before #8542 but this was for 2.7.3 that doesn't have the solution.

I think we need to backport the fix to the 2.7.x branch.

[jeschkies]

@umglurf the backport is up for review #8664

[m1keil]

now amd64 is having the same issue with journal missing in v2.8.0, what's going on? #9060

[jeschkies]

It's not the same issue. We originally excluded journald support for ARM because there was no time to dig into the build system to support it. That's been resolved. The missing support for AMD64 is a simple regression that went unnoticed because no one seems to be using promtail main builds that much for scraping journald.

As annoying as it is that's software development.

[sielicki]

docs at https://grafana.com/docs/loki/latest/send-data/promtail/ need to be updated now that this is resolved:

Currently, Promtail can tail logs from two sources: local log files and the systemd journal (on AMD64 machines only).