Releases: axoflow/axosyslog
axosyslog-charts-0.11.0
AxoSyslog for Kubernetes
axosyslog-charts-0.10.1
AxoSyslog for Kubernetes
axosyslog-4.8.1
4.8.1
This is a bugfix release of AxoSyslog.
AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.
Explore and learn more about the new features in our release announcement blog post.
We provide cloud-ready container images and Helm charts.
Packages are available for Debian and Ubuntu from our APT repository.
RPM packages are available in the Assets section (we’re working on an RPM repository as well, and hope to have it up and running for the next release).
Check out the AxoSyslog documentation for all the details.
Bugfixes
-
Fixed crash around wildard
@include
configuration pragmas when compiled with musl libcThe AxoSyslog container image, for example, was affected by this bug.
(#261)
-
metrics-probe()
: fix disappearing metrics fromstats prometheus
outputmetrics-probe()
metrics became orphaned and disappeared from thesyslog-ng-ctl stats prometheus
output
whenever an ivykis worker stopped (after 10 seconds of inactivity).
(#243) -
syslog-ng-ctl
: fix escaping ofstats prometheus
Metric labels (for example, the ones produced by
metrics-probe()
) may contain control characters, invalid UTF-8 or\
characters. In those specific rare cases, the escaping of thestats prometheus
output was incorrect.
(#224) -
Fixed potential null pointer deref issues
(#216)
Other changes
-
tls()
: expose the key fingerprint of the peer in ${.tls.x509_fp} if
trusted-keys() is used to retain the actual peer identity in received
messages.
(#136) -
network()
,syslog()
sources andsyslog-parser()
: addno-piggyback-errors
flagWith the
no-piggyback-errors
flag ofsyslog-parser()
, the message will not be attributed to AxoSyslog in
case of errors. Actually it retains everything that was present at the time of the parse error,
potentially things that were already extracted.So $MSG remains that was set (potentially the raw message), $HOST may or may not be extracted,
likewise for $PROGRAM, $PID, $MSGID, etc.The error is still indicated via $MSGFORMAT set to "syslog:error".
(#245)
Discord
For a bit more interactive discussion, join our Discord server:
Credits
AxoSyslog is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce AxoSyslog.
Reporting bugs, testing changes, writing code or simply providing
feedback is an important contribution, so please if you are a user
of AxoSyslog, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Attila Szakacs, Balazs Scheidler, Dmitry Levin, Hofi,
László Várady, Szilárd Parrag, shifter
axosyslog-charts-0.10.0
AxoSyslog for Kubernetes
axosyslog-4.8.0
4.8.0
We are excited to announce the first independent release of AxoSyslog.
AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.
Explore and learn more about the new features in our release announcement blog post.
Packages are available for Debian and Ubuntu from our APT repository.
RPM packages are available in the Assets section (we’re working on an RPM repository as well, and hope to have it up and running for the next release).
Check out the AxoSyslog documentation for all the details.
Highlights
Send log messages to Elasticsearch data stream
The elasticsearch-datastream()
destination can be used to feed Elasticsearch data streams.
Example config:
elasticsearch-datastream(
url("https://elastic-endpoint:9200/my-data-stream/_bulk")
user("elastic")
password("ba3DI8u5qX61We7EP748V8RZ")
);
(#178)
Features
-
s3()
: Introduced server side encryption related optionsserver-side-encryption()
andkms-key()
can be used to configure encryption.Currently only
server-side-encryption("aws:kms")
is supported.
Thekms-key()
should be:- an ID of a key
- an alias of a key, but in that case you have to add the alias/prefix
- an ARN of a key
To be able to use the aws:kms encryption the AWS Role or User has to have the following
permissions on the given key:kms:Decrypt
kms:Encrypt
kms:GenerateDataKey
Check this page on why the
kms:Decrypt
is mandatory.Example config:
destination d_s3 { s3( bucket("log-archive-bucket") object-key("logs/syslog") server-side-encryption("aws:kms") kms-key("alias/log-archive") ); };
-
opentelemetry()
,loki()
,bigquery()
destination: Addedheaders()
optionWith this option you can add gRPC headers to each RPC call.
Example config:
opentelemetry( ... headers( "organization" => "Axoflow" "stream-name" => "axo-stream" ) );
(#192)
Bugfixes
-
csv-parser()
: fix escape-backslash-with-sequences dialect on ARM
(#4947) -
csv-parser()
produced invalid output on platforms where char is an unsigned type.
(#4947) -
rate-limit()
: Fixed a crash which occured on a config parse failure.
(#169) -
macros: Fixed a bug which always set certain macros to string type
The affected macros are
$PROGRAM
,$HOST
and$MESSAGE
.
(#162) -
wildcard-file()
: fix crash when a deleted file is concurrently written
(#160) -
disk-buffer()
: fix crash when pipeline initialization failslog_queue_disk_free_method: assertion failed: (!qdisk_started(self->qdisk))
(#128) -
syslog-ng-ctl query
: fix showing Prometheus metrics as unnamed valuesnone.value=726685
(#129) -
syslog-ng-ctl query
: show timestamps and fixg_pattern_spec_match_string
assert
(#129)
Other changes
-
packages/dbld: add support for Ubuntu 24.04 (Noble Numbat)
(#4925) -
syslog-ng-ctl
: do not show orphan metrics forstats prometheus
As the
stats prometheus
command is intended to be used to forward metrics
to Prometheus or any other time-series database, displaying orphaned metrics
should be avoided in order not to insert new data points when a given metric
is no longer alive.In case you are interested in the last known value of orphaned counters, use
thestats
orquery
subcommands.
(#4921) -
bigquery()
,loki()
,opentelemetry()
,cloud-auth()
: C++ modules can be compiled with clangCompiling and using these C++ modules are now easier on FreeBSD and macOS.
(#4933) -
s3()
: new metricsyslogng_output_event_bytes_total
(#4958)
[1] syslog-ng is a trademark of One Identity.
Discord
For a bit more interactive discussion, join our Discord server:
Credits
AxoSyslog is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce AxoSyslog.
Reporting bugs, testing changes, writing code or simply providing
feedback is an important contribution, so please if you are a user
of AxoSyslog, contribute.
We would like to thank the following people for their contribution:
Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Dmitry Levin,
Ferenc HERNADI, Gabor Kozma, Hofi, Ilya Kheifets, Kristof Gyuracz,
László Várady, Máté Őry, Robert Fekete, Szilard Parrag, Wolfram Joost,
shifter
axosyslog-charts-0.9.1
AxoSyslog for Kubernetes
axosyslog-4.7.1
4.7.1
This is the combination of the news entries of 4.7.0
and 4.7.1
.
4.7.1
hotfixed two crashes related to configuration reload.
Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.
Highlights
Collecting Jellyfin logs
The new jellyfin()
source, reads Jellyfin logs from its log file output.
Example minimal config:
source s_jellyfin {
jellyfin(
base-dir("/path/to/my/jellyfin/root/log/dir")
filename-pattern("log_*.log")
);
};
For more details about Jellyfin logging, see:
- https://jellyfin.org/docs/general/administration/configuration/#main-configuration
- https://jellyfin.org/docs/general/administration/configuration/#log-directory
As the jellyfin()
source is based on a wildcard-file()
source, all of the
wildcard-file()
source options are applicable, too.
(#4802)
Collecting *arr logs
Use the newly added *arr()
sources to read various *arr logs:
lidarr()
prowlarr()
radarr()
readarr()
sonarr()
whisparr()
Example minimal config:
source s_radarr {
radarr(
dir("/path/to/my/radarr/log/dir")
);
};
The logging module is stored in the <prefix><module>
name-value pair,
for example: .radarr.module
=> ImportListSyncService
.
The prefix can be modified with the prefix()
option.
(#4803)
Features
-
opentelemetry()
,syslog-ng-otlp()
source: Addedconcurrent-requests()
option.This option configures the maximal number of in-flight gRPC requests per worker.
Setting this value to the range of 10s or 100s is recommended when there are a
high number of clients sending simultaneously.Ideally,
workers() * concurrent-requests()
should be greater or equal to
the number of clients, but this can increase the memory usage.
(#4827) -
loki()
: Support multi-tenancy with the newtenant-id()
option
(#4812) -
s3()
: Added support for authentication from environment.The
access-key()
andsecret-key()
options are now optional,
which makes it possible to use authentication methods originated
from the environment, e.g.AWS_...
environment variables or
credentials files from the~/.aws/
directory.For more info, see:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
(#4881) -
gRPC based drivers: Added
channel-args()
option.Affected drivers are:
bigquery()
destinationloki()
destinationopentelemetry()
source and destinationsyslog-ng-otlp()
source and destination
The
channel-args()
option accepts name-value pairs and sets channel arguments
defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.htmlExample config:
opentelemetry( channel-args( "grpc.loadreporting" => 1 "grpc.minimal_stack" => 0 ) );
(#4827)
-
${TRANSPORT}
macro: Added support for locally created logs.New values are:
- "local+unix-stream"
- "local+unix-dgram"
- "local+file"
- "local+pipe"
- "local+program"
- "local+devkmsg"
- "local+journal"
- "local+afstreams"
- "local+openbsd"
(#4777)
-
tags
: Added new built-in tags that help identifying parse errors.New tags are:
- "message.utf8_sanitized"
- "message.parse_error"
- "syslog.missing_pri"
- "syslog.missing_timestamp"
- "syslog.invalid_hostname"
- "syslog.unexpected_framing"
- "syslog.rfc3164_missing_header"
- "syslog.rfc5424_unquoted_sdata_value"
(#4804)
-
mqtt()
source: Added${MQTT_TOPIC}
name-value pair.It is useful for the cases where
topic()
contains wildcards.Example config:
log { source { mqtt(topic("#")); }; destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); }; };
(#4824)
-
template()
: Added a new template function:$(tags-head)
This template function accepts multiple tag names, and returns the
first one that is set.Example config:
# resolves to "bar" if "bar" tag is set, but "foo" is not template("$(tags-head foo bar baz)")
(#4804)
-
s3()
: Use default AWS URL ifurl()
is not set.
(#4813) -
opentelemetry()
,syslog-ng-otlp()
source: Addedlog-fetch-limit()
option.This option can be used to fine tune the performance. To minimize locking while
moving messages between source and destination side queues, syslog-ng can move
messages in batches. Thelog-fetch-limit()
option sets the maximal size of
the batch moved by a worker. By default it is equal tolog-iw-size() / workers()
.
(#4827) -
dqtool
: add option for truncating (compacting) abandoned disk-buffers
(#4875)
Bugfixes
-
opentelemetry()
: fix crash when an invalid configuration needs to be reverted
(#4910) -
gRPC drivers: fixed a crash when gRPC drivers were used and syslog-ng was reloaded
(#4909) -
opentelemetry()
,syslog-ng-otlp()
source: Fixed a crash.It occurred with multiple
workers()
during high load.
(#4827) -
rename()
: Fixed a bug, which always converted the renamed NV pair to string type.
(#4847) -
With IPv6 disabled, there were linking errors
(#4880)
Metrics
-
http()
: Added a new counter for HTTP requests.It is activated on
stats(level(1));
.Example metrics:
syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1 syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24
(#4805)
-
gRPC based destination drivers: Added gRPC request related metrics.
Affected drivers:
opentelemetry()
syslog-ng-otlp()
bigquery()
loki()
Example metrics:
syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49 syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11
(#4811)
-
New metric to monitor destination reachability
syslogng_output_unreachable
is a bool-like metric, which shows whether a
destination is reachable or not.sum()
can be used to count all unreachable outputs, hence the negated name.It is currently available for the
network()
,syslog()
,unix-*()
destinations, and threaded destinations (http()
,opentelemetry()
,redis()
,
mongodb()
,python()
, etc.).
(#4876) -
destinations: Added "syslogng_output_event_retries_total" counter.
This counter is available for the following destination drivers:
amqp()
bigquery()
http()
and all http based driversjava()
kafka()
loki()
mongodb()
mqtt()
opentelemetry()
python()
and all python based driversredis()
riemann()
smtp()
snmp()
sql()
stomp()
syslog-ng-otlp()
Example metrics:
syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5
(#4807)
-
syslogng_memory_queue_capacity
Shows the capacity (maximum possible size) of each queue.
Note that this metric publisheslog-fifo-size()
, which only limits non-flow-controlled messages.
Messages coming from flow-controlled paths are not limited bylog-fifo-size()
, their corresponding
sourcelog-iw-size()
is the upper limit.
(#4831)
Other changes
-
opentelemetry()
,syslog-ng-otlp()
source: Changed the backpressure behavior.syslog-ng no longer returns
UNAVAILABLE
to the gRPC request, when it cannot forward
the received message because of backpressure. Instead, syslog-ng will block until the
destination can accept more messages.
([#4827](https://github.com/syslog-ng/syslog-ng/pull/...
axosyslog-charts-0.9.1
AxoSyslog for Kubernetes
axosyslog-4.7.0
4.7.0
Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.
Highlights
Collecting Jellyfin logs
The new jellyfin()
source, reads Jellyfin logs from its log file output.
Example minimal config:
source s_jellyfin {
jellyfin(
base-dir("/path/to/my/jellyfin/root/log/dir")
filename-pattern("log_*.log")
);
};
For more details about Jellyfin logging, see:
- https://jellyfin.org/docs/general/administration/configuration/#main-configuration
- https://jellyfin.org/docs/general/administration/configuration/#log-directory
As the jellyfin()
source is based on a wildcard-file()
source, all of the
wildcard-file()
source options are applicable, too.
(#4802)
Collecting *arr logs
Use the newly added *arr()
sources to read various *arr logs:
lidarr()
prowlarr()
radarr()
readarr()
sonarr()
whisparr()
Example minimal config:
source s_radarr {
radarr(
dir("/path/to/my/radarr/log/dir")
);
};
The logging module is stored in the <prefix><module>
name-value pair,
for example: .radarr.module
=> ImportListSyncService
.
The prefix can be modified with the prefix()
option.
(#4803)
Features
-
opentelemetry()
,syslog-ng-otlp()
source: Addedconcurrent-requests()
option.This option configures the maximal number of in-flight gRPC requests per worker.
Setting this value to the range of 10s or 100s is recommended when there are a
high number of clients sending simultaneously.Ideally,
workers() * concurrent-requests()
should be greater or equal to
the number of clients, but this can increase the memory usage.
(#4827) -
loki()
: Support multi-tenancy with the newtenant-id()
option
(#4812) -
s3()
: Added support for authentication from environment.The
access-key()
andsecret-key()
options are now optional,
which makes it possible to use authentication methods originated
from the environment, e.g.AWS_...
environment variables or
credentials files from the~/.aws/
directory.For more info, see:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
(#4881) -
gRPC based drivers: Added
channel-args()
option.Affected drivers are:
bigquery()
destinationloki()
destinationopentelemetry()
source and destinationsyslog-ng-otlp()
source and destination
The
channel-args()
option accepts name-value pairs and sets channel arguments
defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.htmlExample config:
opentelemetry( channel-args( "grpc.loadreporting" => 1 "grpc.minimal_stack" => 0 ) );
(#4827)
-
${TRANSPORT}
macro: Added support for locally created logs.New values are:
- "local+unix-stream"
- "local+unix-dgram"
- "local+file"
- "local+pipe"
- "local+program"
- "local+devkmsg"
- "local+journal"
- "local+afstreams"
- "local+openbsd"
(#4777)
-
tags
: Added new built-in tags that help identifying parse errors.New tags are:
- "message.utf8_sanitized"
- "message.parse_error"
- "syslog.missing_pri"
- "syslog.missing_timestamp"
- "syslog.invalid_hostname"
- "syslog.unexpected_framing"
- "syslog.rfc3164_missing_header"
- "syslog.rfc5424_unquoted_sdata_value"
(#4804)
-
mqtt()
source: Added${MQTT_TOPIC}
name-value pair.It is useful for the cases where
topic()
contains wildcards.Example config:
log { source { mqtt(topic("#")); }; destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); }; };
(#4824)
-
template()
: Added a new template function:$(tags-head)
This template function accepts multiple tag names, and returns the
first one that is set.Example config:
# resolves to "bar" if "bar" tag is set, but "foo" is not template("$(tags-head foo bar baz)")
(#4804)
-
s3()
: Use default AWS URL ifurl()
is not set.
(#4813) -
opentelemetry()
,syslog-ng-otlp()
source: Addedlog-fetch-limit()
option.This option can be used to fine tune the performance. To minimize locking while
moving messages between source and destination side queues, syslog-ng can move
messages in batches. Thelog-fetch-limit()
option sets the maximal size of
the batch moved by a worker. By default it is equal tolog-iw-size() / workers()
.
(#4827) -
dqtool
: add option for truncating (compacting) abandoned disk-buffers
(#4875)
Bugfixes
-
opentelemetry()
,syslog-ng-otlp()
source: Fixed a crash.It occurred with multiple
workers()
during high load.
(#4827) -
rename()
: Fixed a bug, which always converted the renamed NV pair to string type.
(#4847) -
With IPv6 disabled, there were linking errors
(#4880)
Metrics
-
http()
: Added a new counter for HTTP requests.It is activated on
stats(level(1));
.Example metrics:
syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2 syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1 syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24
(#4805)
-
gRPC based destination drivers: Added gRPC request related metrics.
Affected drivers:
opentelemetry()
syslog-ng-otlp()
bigquery()
loki()
Example metrics:
syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49 syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11
(#4811)
-
New metric to monitor destination reachability
syslogng_output_unreachable
is a bool-like metric, which shows whether a
destination is reachable or not.sum()
can be used to count all unreachable outputs, hence the negated name.It is currently available for the
network()
,syslog()
,unix-*()
destinations, and threaded destinations (http()
,opentelemetry()
,redis()
,
mongodb()
,python()
, etc.).
(#4876) -
destinations: Added "syslogng_output_event_retries_total" counter.
This counter is available for the following destination drivers:
amqp()
bigquery()
http()
and all http based driversjava()
kafka()
loki()
mongodb()
mqtt()
opentelemetry()
python()
and all python based driversredis()
riemann()
smtp()
snmp()
sql()
stomp()
syslog-ng-otlp()
Example metrics:
syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5
(#4807)
-
syslogng_memory_queue_capacity
Shows the capacity (maximum possible size) of each queue.
Note that this metric publisheslog-fifo-size()
, which only limits non-flow-controlled messages.
Messages coming from flow-controlled paths are not limited bylog-fifo-size()
, their corresponding
sourcelog-iw-size()
is the upper limit.
(#4831)
Other changes
-
opentelemetry()
,syslog-ng-otlp()
source: Changed the backpressure behavior.syslog-ng no longer returns
UNAVAILABLE
to the gRPC request, when it cannot forward
the received message because of backpressure. Instead, syslog-ng will block until the
destination can accept more messages.
(#4827) -
opentelemetry()
,syslog-ng-otlp()
source:log-iw-size()
is now split between workers.
(#4827) -
APT packages: Dropped Debian Buster support.
Old packages are still available, but new syslog-ng versions will not
be available on Debian Buster
(#4840) -
dbld
: AlmaLinux 8 support
([#49...
axosyslog-charts-0.9.0
AxoSyslog for Kubernetes