Add journald receiver #2332

djaglowski · 2021-02-11T19:04:15Z

Dependent on #2268

wph95 · 2021-02-23T08:37:13Z

could assign this one to me if no one is working on it.

I saw OTel-log-collection supported journaled_input, which seems to use internal/stanze/createLogsReceiver can resolved this issue

djaglowski · 2021-02-25T16:18:12Z

Thank you @wph95. We'll happily accept the help.

You've got the idea exactly right. internal/stanza should make it fairly easy to create the new receiver using journald_input, which will look very similar to filelogreceiver.

sumo-drosiek · 2021-03-09T13:27:32Z

@wph95 Do you need some help?

I made some research around using journald_input and this plugin requires journalctl binary to work. As this was my major concern I prepared Dockerfile which includes journalctl. This is made on the top of debian and make docker-otelcontribcol

FROM debian:10.8 as journal
RUN apt update
RUN apt install -y systemd

FROM otelcontribcol:latest
COPY --from=journal /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
COPY --from=journal /lib/systemd/libsystemd-shared-241.so /lib/systemd/libsystemd-shared-241.so
COPY --from=journal /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
COPY --from=journal /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/librt.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libcap.so.2 /lib/x86_64-linux-gnu/libcap.so.2
COPY --from=journal /usr/lib/x86_64-linux-gnu/libacl.so.1 /usr/lib/x86_64-linux-gnu/libacl.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libcryptsetup.so.12 /lib/x86_64-linux-gnu/libcryptsetup.so.12
COPY --from=journal /lib/x86_64-linux-gnu/libgcrypt.so.20 /lib/x86_64-linux-gnu/libgcrypt.so.20
COPY --from=journal /usr/lib/x86_64-linux-gnu/libip4tc.so.0 /usr/lib/x86_64-linux-gnu/libip4tc.so.0
COPY --from=journal /usr/lib/x86_64-linux-gnu/libkmod.so.2 /usr/lib/x86_64-linux-gnu/libkmod.so.2
COPY --from=journal /lib/x86_64-linux-gnu/libmount.so.1 /lib/x86_64-linux-gnu/libmount.so.1
COPY --from=journal /usr/lib/x86_64-linux-gnu/libseccomp.so.2 /usr/lib/x86_64-linux-gnu/libseccomp.so.2
COPY --from=journal /lib/x86_64-linux-gnu/libselinux.so.1 /lib/x86_64-linux-gnu/libselinux.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libidn.so.11 /lib/x86_64-linux-gnu/libidn.so.11
COPY --from=journal /lib/x86_64-linux-gnu/liblzma.so.5 /lib/x86_64-linux-gnu/liblzma.so.5
COPY --from=journal /usr/lib/x86_64-linux-gnu/liblz4.so.1 /usr/lib/x86_64-linux-gnu/liblz4.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libblkid.so.1 /lib/x86_64-linux-gnu/libblkid.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libpthread.so.0
COPY --from=journal /usr/lib/x86_64-linux-gnu/libattr.so.1 /usr/lib/x86_64-linux-gnu/libattr.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libuuid.so.1 /lib/x86_64-linux-gnu/libuuid.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
COPY --from=journal /usr/lib/x86_64-linux-gnu/libssl.so.1.1 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
COPY --from=journal /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
COPY --from=journal /usr/lib/x86_64-linux-gnu/libargon2.so.1 /usr/lib/x86_64-linux-gnu/libargon2.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so.2
COPY --from=journal /usr/lib/x86_64-linux-gnu/libjson-c.so.3 /usr/lib/x86_64-linux-gnu/libjson-c.so.3
COPY --from=journal /lib/x86_64-linux-gnu/libgpg-error.so.0 /lib/x86_64-linux-gnu/libgpg-error.so.0
COPY --from=journal /lib/x86_64-linux-gnu/libpcre.so.3 /lib/x86_64-linux-gnu/libpcre.so.3
COPY --from=journal /lib/x86_64-linux-gnu/libudev.so.1 /lib/x86_64-linux-gnu/libudev.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6

I believe this can be useful

wph95 · 2021-03-10T02:05:57Z

@wph95 Do you need some help?

I made some research around using journald_input and this plugin requires journalctl binary to work. As this was my major concern I prepared Dockerfile which includes journalctl. This is made on the top of debian and make docker-otelcontribcol

FROM debian:10.8 as journal
RUN apt update
RUN apt install -y systemd

FROM otelcontribcol:latest
COPY --from=journal /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
COPY --from=journal /lib/systemd/libsystemd-shared-241.so /lib/systemd/libsystemd-shared-241.so
COPY --from=journal /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
COPY --from=journal /lib/x86_64-linux-gnu/librt.so.1 /lib/x86_64-linux-gnu/librt.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libcap.so.2 /lib/x86_64-linux-gnu/libcap.so.2
COPY --from=journal /usr/lib/x86_64-linux-gnu/libacl.so.1 /usr/lib/x86_64-linux-gnu/libacl.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libcryptsetup.so.12 /lib/x86_64-linux-gnu/libcryptsetup.so.12
COPY --from=journal /lib/x86_64-linux-gnu/libgcrypt.so.20 /lib/x86_64-linux-gnu/libgcrypt.so.20
COPY --from=journal /usr/lib/x86_64-linux-gnu/libip4tc.so.0 /usr/lib/x86_64-linux-gnu/libip4tc.so.0
COPY --from=journal /usr/lib/x86_64-linux-gnu/libkmod.so.2 /usr/lib/x86_64-linux-gnu/libkmod.so.2
COPY --from=journal /lib/x86_64-linux-gnu/libmount.so.1 /lib/x86_64-linux-gnu/libmount.so.1
COPY --from=journal /usr/lib/x86_64-linux-gnu/libseccomp.so.2 /usr/lib/x86_64-linux-gnu/libseccomp.so.2
COPY --from=journal /lib/x86_64-linux-gnu/libselinux.so.1 /lib/x86_64-linux-gnu/libselinux.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libidn.so.11 /lib/x86_64-linux-gnu/libidn.so.11
COPY --from=journal /lib/x86_64-linux-gnu/liblzma.so.5 /lib/x86_64-linux-gnu/liblzma.so.5
COPY --from=journal /usr/lib/x86_64-linux-gnu/liblz4.so.1 /usr/lib/x86_64-linux-gnu/liblz4.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libblkid.so.1 /lib/x86_64-linux-gnu/libblkid.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libpthread.so.0
COPY --from=journal /usr/lib/x86_64-linux-gnu/libattr.so.1 /usr/lib/x86_64-linux-gnu/libattr.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libuuid.so.1 /lib/x86_64-linux-gnu/libuuid.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
COPY --from=journal /usr/lib/x86_64-linux-gnu/libssl.so.1.1 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
COPY --from=journal /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
COPY --from=journal /usr/lib/x86_64-linux-gnu/libargon2.so.1 /usr/lib/x86_64-linux-gnu/libargon2.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so.2
COPY --from=journal /usr/lib/x86_64-linux-gnu/libjson-c.so.3 /usr/lib/x86_64-linux-gnu/libjson-c.so.3
COPY --from=journal /lib/x86_64-linux-gnu/libgpg-error.so.0 /lib/x86_64-linux-gnu/libgpg-error.so.0
COPY --from=journal /lib/x86_64-linux-gnu/libpcre.so.3 /lib/x86_64-linux-gnu/libpcre.so.3
COPY --from=journal /lib/x86_64-linux-gnu/libudev.so.1 /lib/x86_64-linux-gnu/libudev.so.1
COPY --from=journal /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6

I believe this can be useful

Thank you for your helpful message. I'm waiting for the Syslog receiver to finish and start doing the journal, and this tip is very useful for me.

by the way, I'm thinking about opentelemetry-log-collection use journalctl binary directly maybe is not a good way, maybe use https://github.com/coreos/go-systemd/ is a good way. (will discuss it in opentelemetry-log-collection

tigrannajaryan · 2021-03-19T16:20:59Z

by the way, I'm thinking about opentelemetry-log-collection use journalctl binary directly maybe is not a good way, maybe use https://github.com/coreos/go-systemd/ is a good way. (will discuss it in opentelemetry-log-collection

Yes, preferably we should not have a dependency on an external binary. (Is journalctl guaranteed to be available on any system that uses journald?)

However, we need to consider whether https://github.com/coreos/go-systemd/ is mature enough and how much more work it would be to use it instead of the existing opentelemetry-log-collection solution.

ttomsu · 2021-06-18T13:10:16Z

Any progress on this? Would love be able to pull in journald entries into our logging solution!

djaglowski · 2021-06-18T16:09:12Z

@ttomsu, thanks for bringing this up. I think the next steps are to address Tigran's points.

If we end up depending on journalctl, the effort should be fairly straightforward.

To that end:

Is journalctl guaranteed to be available on any system that uses journald?

I believe the short answer is Yes. journald and journalctl are both part of systemd. You can see journald* files and journalctl.c in the same source directory here. Also in this directory is a build file that appears to build these into separate executables.

we need to consider whether https://github.com/coreos/go-systemd/ is mature enough and how much more work it would be to use it instead of the existing opentelemetry-log-collection solution

go-systemd apparently requires CGO. I believe this is not acceptable for the collector. Is that correct?

jsirianni · 2021-06-18T16:09:40Z

We bundle systemd (which includes journalctl) into Stanza image.

Before that, we required CGO. The decision was made that relying on a binary (inside the docker image) was preferable. Image size is much larger using this method.

ttomsu · 2021-07-22T15:56:23Z

Friendly ping - we have users interested in this functionality.

rockb1017 · 2021-08-31T17:05:47Z

we will be working on this.

rockb1017 · 2021-09-02T23:03:06Z

We are implementing it with journalctl binary dependency in the image.
@wph95 Thanks for providing that Dockerfile. Really helpful.
I just needed to add this line
COPY --from=journal /bin/journalctl /bin/journalctl
to your Dockerfile to make it work.
Should I update the dockerfile with this PR?
this is the size difference between 2 images.

otelcontribcol                base        b812fa87fe32   23 hours ago     133MB
otelcontribcol                latest      863c7bdbb7ef   13 minutes ago   148MB

latest tag is the one with journalctl added.

gillg · 2021-09-09T08:11:32Z

Hello, like said in the associated PR, maybe an alternative to the dirty docker trick could be to propose install systemd-journal-remote to the user as we need to install rsyslog to use syslog receiver ?
In this case instead of read local files with journalctl we bind an HTTP endpoint to receive the journal content in a pre-translated format (json probably).

What do you think about that ? The collector stay platform / OS independant, and a good help ton configure the journal-gateway service could be enougth.
Maybe this could be integrated as input of log-receiver project ?

EDIT: man of journal-gateway https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html

sumo-drosiek · 2021-09-09T11:03:29Z

@gillg What in case, user cannot/doesn't want to modify systemd configuration, for example on kubernetes nodes?
adding complementary pod/container which will read systemd logs and forward them to OTC instance?

gillg · 2021-09-09T11:26:07Z

@gillg What in case, user cannot/doesn't want to modify systemd configuration, for example on kubernetes nodes?
adding complementary pod/container which will read systemd logs and forward them to OTC instance?

It can be a solution, seems a little bit tricky.
But in my opinion, if someone care about host logs in journald he's probably not a simple kubernetes user. And in this case if he wants collect system logs (for compliance, audit or diag) it seems legit to fine tune systemd config. So add this gateway on the base node template seems acceptable no ?

Else, we could provide help with a dedicated daemonset init container, which mount the full host FS (like a node exporter does), then chroot on it to install the journal-gateway service, configure it to point on otel collector, and terminate. It's the same design than kube makes to install CNI on every new nodes. Any other ideas are welcome.

jsirianni · 2021-09-09T13:58:07Z

Journald is used for many Kubernetes deployments, for example:

Kubelet logs
OpenShift uses Journald by default for all container logs

gillg · 2021-10-06T17:17:37Z

Regarding the final feature I don't see any missing things if we use the remote journalctl interface instead of the binary journalctl.

If you want expose journald in a readable format, journalctl is the way localy and journald-gateway is the way remotely.

Final possible solution :

We use a sidecar pattern container (called "journald-gateway" as exemple) built with something like that :

FROM debian:stable-slim

RUN apt update && apt install -y systemd-journal-remote && apt clean
# For RHEL : yum install systemd-journal-gateway

EXPOSE 19531
 
# Doc https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html
ENTRYPOINT ["/lib/systemd/systemd-journal-gatewayd"]
# For RHEL : /usr/lib/systemd/systemd-journal-gatewayd

CMD ["--directory=/host/var/log/journal"]

We create it on host as sidecar with docker run -v /var/log/journal/:/host/var/log/journal/ -ti journald-gateway
an alternative can be an init container to install journal-gatewayd on host with chroot (a little bit dirty)
Otel container can reach it with http://journald-gateway:19531/entries?follow and http://journald-gateway:19531/machine to build resources attributes eventualy
On otel collector we use a config like :

receiver:
  journald:
      mode: gateway #in mode gateway endpoint is needed, in eventual mode local, directory and files can be used instead
      endpoint: http://journald-gateway:19531/
      units:
        - ssh
        - kublet
        - dockerd
      priority: info
# directory and files parameters can be passed to the journald-gateway container at startup but not at otel-collector level

All documentation here https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html#Supported%20URLs
We can eventualy have only one collector for multiple nodes if we don't want the sidecar pattern.

This imply to contribute on stanza side to change a little the way to call journalctl (web endpoint instead of subprocess, but with equivalent features)

Finally... If a user really want to use journalctl in otel container, he can create and maintain its own layer with all COPY mentioned above. But otel container remains clean and minimalist. (We don't even have sh or curl on it... So why we should have journalctl if viable solution for the same feature exist ?)

dmitryax · 2021-10-07T17:02:53Z

This imply to contribute on stanza side to change a little the way to call journalctl (web endpoint instead of subprocess, but with equivalent features)

@gillg @djaglowski this will give us the checkpoints support that is available in the stanza library, right?

I like the suggested approach, but I think that being able to store checkpoints should be a requirement to avoid data duplication/loss.

djaglowski · 2021-10-07T17:19:55Z

@dmitryax It appears that journald-gateway supports a Range Header, which gives us a mechanism for remote checkpointing. (And to be clear, local checkpointing w/ journalctl is already supported in this library.)

dmitryax · 2021-10-07T17:29:42Z

@djaglowski Nice! I think we can introduce the first version relying on journalctl, then we can add an option to use journald-gateway in a separate PR. What do you think?

gillg · 2021-10-07T17:58:11Z

@dmitryax assuming we skip the changes in Dockerfile and if a user want use it he creates his own image ? In my opinion it makes sense for a beginning :)

gillg · 2021-10-07T20:32:42Z

By the way... @tigrannajaryan do you think that a sidecar image with journald gateway should be part of opentelemtry "tools" and hosted in otel docker repo ?
An associated build step in the makefile or a dedicated (very little...) could be added to update it sometimes.
The image should be multi architecture in the meantime.

Link to tracking Issue: #2332 Testing: Unit tests are included Documentation: Journald receiver README.md is included

tigrannajaryan · 2021-10-13T17:40:09Z

Resolved by #5160

If there is an additional interest in discussing the docker packaging please open a separate issue.

Resolve #2332

This was referenced Feb 11, 2021

Add basic log collection capabilities #2264

Closed

Add mapstructure support to journald_input operator configuration open-telemetry/opentelemetry-log-collection#17

Closed

djaglowski added the spec:logs label Feb 15, 2021

djaglowski assigned wph95 Feb 25, 2021

tigrannajaryan added this to the Basic Logs Support milestone Mar 3, 2021

djaglowski unassigned wph95 Jun 18, 2021

tigrannajaryan assigned rockb1017 Aug 31, 2021

rockb1017 mentioned this issue Sep 2, 2021

adding journalctl to image signalfx/splunk-otel-collector#715

Closed

luckyj5 mentioned this issue Sep 8, 2021

Add journald receiver #5160

Merged

tigrannajaryan pushed a commit that referenced this issue Oct 13, 2021

Add journald receiver (#5160)

fab94ea

Link to tracking Issue: #2332 Testing: Unit tests are included Documentation: Journald receiver README.md is included

tigrannajaryan closed this as completed Oct 13, 2021

gillg mentioned this issue Dec 9, 2021

Journaldreceiver - Add documentation and build an "official" docker image for journald-gateway #6672

Closed

ljmsc referenced this issue in ljmsc/opentelemetry-collector-contrib Feb 21, 2022

Correct getting started docs typo (#2333)

4ba964b

Resolve #2332

dbason mentioned this issue Mar 5, 2023

New controller for OTEL collectors rancher/opni#1051

Closed

3 tasks

pecigonzalo mentioned this issue Apr 17, 2024

Contrib image: journaldreceiver receiver is missing journalctl open-telemetry/opentelemetry-collector-releases#462

Open

sumo-drosiek mentioned this issue Apr 26, 2024

[journaldreceiver]: implement golang based reader for journaldreceiver #32711

Open

cmacknz mentioned this issue Jun 6, 2024

GA support for reading from journald elastic/beats#37086

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add journald receiver #2332

Add journald receiver #2332

djaglowski commented Feb 11, 2021

wph95 commented Feb 23, 2021

djaglowski commented Feb 25, 2021 •

edited

Loading

sumo-drosiek commented Mar 9, 2021

wph95 commented Mar 10, 2021

tigrannajaryan commented Mar 19, 2021

ttomsu commented Jun 18, 2021

djaglowski commented Jun 18, 2021

jsirianni commented Jun 18, 2021 •

edited

Loading

ttomsu commented Jul 22, 2021

rockb1017 commented Aug 31, 2021

rockb1017 commented Sep 2, 2021

gillg commented Sep 9, 2021 •

edited

Loading

sumo-drosiek commented Sep 9, 2021 •

edited

Loading

gillg commented Sep 9, 2021

jsirianni commented Sep 9, 2021

gillg commented Oct 6, 2021

dmitryax commented Oct 7, 2021

djaglowski commented Oct 7, 2021

dmitryax commented Oct 7, 2021 •

edited

Loading

gillg commented Oct 7, 2021

gillg commented Oct 7, 2021

tigrannajaryan commented Oct 13, 2021

Add journald receiver #2332

Add journald receiver #2332

Comments

djaglowski commented Feb 11, 2021

wph95 commented Feb 23, 2021

djaglowski commented Feb 25, 2021 • edited Loading

sumo-drosiek commented Mar 9, 2021

wph95 commented Mar 10, 2021

tigrannajaryan commented Mar 19, 2021

ttomsu commented Jun 18, 2021

djaglowski commented Jun 18, 2021

jsirianni commented Jun 18, 2021 • edited Loading

ttomsu commented Jul 22, 2021

rockb1017 commented Aug 31, 2021

rockb1017 commented Sep 2, 2021

gillg commented Sep 9, 2021 • edited Loading

sumo-drosiek commented Sep 9, 2021 • edited Loading

gillg commented Sep 9, 2021

jsirianni commented Sep 9, 2021

gillg commented Oct 6, 2021

dmitryax commented Oct 7, 2021

djaglowski commented Oct 7, 2021

dmitryax commented Oct 7, 2021 • edited Loading

gillg commented Oct 7, 2021

gillg commented Oct 7, 2021

tigrannajaryan commented Oct 13, 2021

djaglowski commented Feb 25, 2021 •

edited

Loading

jsirianni commented Jun 18, 2021 •

edited

Loading

gillg commented Sep 9, 2021 •

edited

Loading

sumo-drosiek commented Sep 9, 2021 •

edited

Loading

dmitryax commented Oct 7, 2021 •

edited

Loading