07 Jul 06:53

github-actions

5a73364

syslog-ng-3.33.1

3.33.1

Highlights

MQTT destination

The new mqtt() destination can be used to publish messages using the MQTT protocol.
Currently MQTT 3.1.1 and 3.1 are supported.

Supported transports: tcp, ws.

Example config:
```
destination {
  mqtt(
    address("tcp://localhost:1883"),
    topic("syslog/$HOST"),
    fallback-topic("syslog/fallback")
  );
};
```
Note: MQTT 5.0 and TLS (ssl://, wss://) are currently not supported.
(#3703)
discord() destination

syslog-ng now has a webhook-based Discord destination.
Example usage:
```
destination {
  discord(url("https://discord.com/api/webhooks/x/y"));
};
```
The following options can be used to customize the destination further:

avatar-url(), username("$HOST-bot"), tts(true), template("${MSG:-[empty message]}").
(#3717)

Features

kafka-c: batching support in case of sync-send(yes)
```
kafka-c(
 bootstrap-server("localhost:9092")
 topic("syslog-ng")
 sync-send(yes)
 batch-lines(10)
 batch-timeout(10000)
);
```
Note1: batch-lines are accepted in case of sync-send(no), but no batching is done.
Note2: messages are still sent one at a time to kafka, the batch yields multiple message per transaction.
(#3699)
kafka-c: sync-send(yes) enables synchronous message delivery, reducing the possibility of message loss.
```
kafka-c(
  bootstrap-server("localhost:9092")
  topic("syslog-ng")
  sync-send(yes)
);
```
Warning: this option also reduces significantly the performance of kafka-c driver.
(#3681)
disk-buffer: Now we optimize the file truncating frequency of disk-buffer.

The new behavior saves IO time, but loses some disk space, which is configurable with a new option.
The new option in the config is settable at 2 places:
- truncate-size-ratio() in the disk-buffer() block, which affects the given disk-buffer.
- disk-buffer(truncate-size-ratio()) in the global options block, which affects every disk-buffer
  which did not set truncate-size-ratio() itself.
  The default value is 0.01, which operates well with most disk-buffers.
If the possible size reduction of the truncation does not reach truncate-size-ratio() x disk-buf-size(),
we do not truncate the disk-buffer.

To completely turn off truncating (maximal disk space loss, maximal IO time saved) set truncate-size-ratio(1),
or to mimic the old behavior (minimal disk space loss, minimal IO time saved) set truncate-size-ratio(0).
(#3689)

Bugfixes

syslog-format: fixing the check-hostname(yes|no) option

The check-hostname(yes|no) option detected every value as invalid, causing a parse error when enabled.
(#3690)
disk-buffer(): fix crash when switching between disk-based and memory queues

When a disk-buffer was removed from the configuration and the new config was
applied by reloading syslog-ng, a crash occurred.
(#3700)
logpath: Fixed a message write protection bug, where message modifications (rewrite rules, parsers, etc.)
leaked through preceding path elements. This may have resulted not only in unwanted/undefined message modification,
but in certain cases crash as well.
(#3708)
mongodb(): fix crash with older mongo-c-driver versions

syslog-ng crashed (was aborted) when the mongodb() destination was used with
older mongo-c-driver versions (< v1.11.0).
(#3677)
java(): fix debug logging of Java-based destinations

Java debug logging was not enabled previously when syslog-ng was started in debug/trace mode. This has been fixed.
(#3679)
kafka-c: fixed a hang during shutdown/reload, when multiple workers is used (workers() option is set to 2 or higher) and the librdkafka internal queue is filled.
(error message was kafka: failed to publish message; topic='test-topic', error='Local: Queue full')
(#3711)

Packaging

kafka: minimum version of librdkafka is changed from 1.0.0 to 1.1.0
(#3706)
configure: now supporting python with two digit minor version
(#3713)

Other changes

kafka: removed some deprecated options: client-lib-dir(), option(), template(), kafka-bootstrap-servers()
(#3698)
kafka: properties-file() option is deprecated. Please list the librdkafka properties in the config() option in syslog-ng's configuration.
(#3698)
smtp(): libesmtp is now detected via pkg-config
(#3669)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler,
Balázs Barkó, Benedek Cserhati, Gabor Nagy, L4rS6, Laszlo Budai, Laszlo Szemere,
LittleFish33, László Várady, Norbert Takacs, Peter Czanik, Peter Kokai,
Todd C. Miller, Tomáš Mózes, Zoltan Pallagi

Assets 3

07 May 14:19

github-actions

syslog-ng-3.32.1

b223781

syslog-ng-3.32.1

3.32.1

Highlights

mongodb(): add workers() support (multi-threaded connection pooling)

The MongoDB driver now supports the workers() option, which specifies the
number of parallel workers to be used.
Workers are based on the connection pooling feature of the MongoDB C library.

This increases the throughput of the MongoDB destination driver.

Example:
```
destination {
  mongodb(
    uri("mongodb://hostA,hostB/syslog?replicaSet=my_rs&wtimeoutMS=10000&socketTimeoutMS=10000&connectTimeoutMS=10000&serverSelectionTimeoutMS=5000")
    collection("messages")
    workers(8)
  );
};
```
(#3621)
mongodb(): template support for the collection() option

The collection() option of the MongoDB destination driver now accepts
templates, for example:
```
destination {
  mongodb(
    uri("mongodb://host/syslog")
    collection("${HOST}_messages")
  );
};
```
(#3621)

Features

time-reopen: Support the time-reopen() option on the driver level for the following drivers:
- sources: example-diskq-source, python-fetcher
- destinations: amqp, example-destination, file, http, mongodb, network, pipe,
  program, pseudofile, python, redis, riemann, smtp, sql, stomp,
  syslog, tcp, tcp6, udp, udp6, unix-dgram, unix-stream, usertty
  (#3585)
csv-parser(): add drop-invalid() option along with the already existing
flag with the same name. This is to improve the consistency of the
configuration language.
(#3547)
usertty() destination: Support changing the terminal disable timeout with the time-reopen() option.
Default timeout change to 60 from 600. If you wish to use the old 600 timeout, add time-reopen(600)
to your config in the usertty() driver.
(#3585)
syslog-parser(): add a new drop-invalid() option that allows the use of
syslog-parser() in if statements. Normally a syslog-parser() injects an
error message instead of failing.
(#3565)

Bugfixes

date-parser: if the timestamp pattern did not covered a field (for example seconds) that field had undefined value

The missing fields are initialized according to the following rules:
1. missing all fields -> use current date
2. only miss year -> guess year based on current year and month (current year, last year or next year)
3. the rest of the cases don't make much sense, so zero initialization of the missing field makes sense. And the year is initialized to the current one.
  (#3615)
Fix compilation issues on OpenBSD

syslog-ng can now be compiled on OpenBSD.
(#3661)
loggen: debug message printed wrong plugin name (ssl-plugin instead of socket_plugin)
(#3624)
tls: fixup EOF detection issue in tls (before 3.0 version)

syslog-ng error message:
"I/O error occurred while reading; fd='13', error='Success (0)'"
(#3618)
kafka: the config() block couldn't contain option that is already a keyword in syslog-ng (example: retries)
(#3658)
templates: fixed error reporting when invalid templates were specified

The amqp(), file() destination, sql(), stomp(), pdbtool, and
graphite() plugins had template options that did not report errors at startup
when invalid values were specified.
(#3660)

Packaging

bison: minimum version of bison is bumped to 3.7.6
(#3547)
java-modules: the minimum version of gradle changed from 2.2 to 3.4
(#3645)
light: add to the release tarball
(#3613)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Attila Szakacs, Balazs Scheidler,
Gabor Nagy, Janos SZIGETVARI, Laszlo Budai, Laszlo Szemere,
LittleFish33, László Várady, Ming Liu, Norbert Takacs, Peter Kokai,
Todd C. Miller, Yi Fan Yu, Zoltan Pallagi

Assets 3

18 Mar 13:57

github-actions

syslog-ng-3.31.2

bd57b4e

syslog-ng-3.31.2

3.31.2

Bugfixes

syslog-parser(): fix a potential crash in case parsing the message
fails and tags are already applied to the message.

Packaging

python2: Direct python2 support is dropped, we no longer test it in our CI.
No python2 related source codes were removed as for now, but we do not
guarantee that it will work in the future.

(#3603)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler,
Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady,
Norbert Takacs, Peter Kokai, Ryan Faircloth, Zoltan Pallagi

Assets 3

04 Mar 13:38

github-actions

syslog-ng-3.31.1

d50df8e

syslog-ng-3.31.1

3.31.1

Highlights

fortigate-parser(): new parser to parse fortigate logs

Example:
```
log {
  source { network(transport("udp") flags(no-parse)); };
  parser { fortigate-parser(); };
  destination { };
};
```
An adapter to automatically recognize fortigate logs in app-parser() has
also been added.
(#3536)
patterndb: Added OPTIONALSET parser. It works the same as SET, but continues, even if none of the
characters is found.
(#3540)

Features

syslog-parser(): add no-header flag to tell syslog-ng to parse only the
PRI field of an incoming message, everything else is just put into $MSG.
(#3538)
set-pri(): this new rewrite operation allows you to change the PRI value
of a message based on the string directly parsed out of a syslog header.
(#3546)

telegram: option to send silent message

Example:

destination { telegram(bot-id(...) chat-id(...) disable_notification(true)); };

(#3558)

app-parser(): added automatic classification & parsing for project Lumberjack/Mitre CEE formatted logs
(#3569)
diskq: if the dir() path provided by the user does not exists, syslog-ng creates the path with the same permission as the running instance
(#3550)

Bugfixes

network(), syslog() destinations: fix reconnection timer when DNS lookups are slow

After a long-lasting DNS query, syslog-ng did not wait the specified time (time_reopen())
before reconnecting to a destination. This has been fixed.
(#3526)
cmake: minor fixes
(#3523)
stats-level(): fix processing the changes in the stats-level() global
option: changes in stats-level() were not reflected in syslog
facility/severity related and message tag related counters after first
configuration reload. These counters continued to operate according to the
value of stats-level() at the first reload.
(#3561)
date-parser(): fix hour-only timezone parsing

If the timestamp contains a short timezone offset (e.g. hours only), the
recent change in strptime() introduces an error, it interprets those
numbers as minutes instead of hours. For example: Jan 16 2019 18:23:12 +05
(#3555)
loggen: fix undefined timeout while connecting to network sources (glib < 2.32)

When compiling syslog-ng with old glib versions (< 2.32), loggen could fail due a timeout bug.
This has been fixed.
(#3504)
grouping-by(): fix deadlock when context expires

In certain cases, the grouping-by() parser could get stuck when a message
context expired, causing a deadlock in syslog-ng.

This has been fixed.
(#3515)
date-parser: Fixed a crash, which occured sometimes when %z was used.
(#3553)
date-parser: %z. We no longer ignore daylight saving time when calculating the GMT offset.
(#3553)
kafka-c: fix a double LogMessage acknowledgement bug, which can cause crash with segmentation fault or exit with sigabrt. The issue affects both flow-controlled and non-flow-controlled log paths and it's triggered in case previously published messages failed to be delivered to Kafka.
(#3583)
python destination: Fixed a rare crash during reload.
(#3568)
date-parser(): fix non-mandatory parsing of timezone name

When %Z is used, the presence of the timezone qualifier is not mandatory,
so don't fail that case.
(#3555)
wildcard-file(): fix infrequent crash when file renamed/recreated

The wildcard-file source crashed when a file being processed was replaced by
a new one on the same path (renamed, deleted+recreated, rotated, etc.).
(#3513)
Remove the no-parse flag in system() source from FreeBSD kernel
messages, so the message header is no more part of the message.
(#3586)
Fix abort on macOS Big Sur

A basic subset of syslog-ng's functionality now works on the latest macOS version.
(#3522)
affile: Fix improper initialization in affile and LogWriter to avoid memory leak when reloading
(#3574)
udp destination: Fixed a bug, where the packet's checksum was not calculated,
when spoof-source(yes) and ip-protocol(6) were set.
(#3528)
python: fix LogMessage.keys() listing non-existenting keys and duplicates
(#3557)

Packaging

Simplify spec file by removing obsolete technologies:
- remove RHEL 6 support
- remove Python 2 support
- keep Java support, but remove Java-based drivers (HDFS, etc.)
  (#3587)
libnet: Minimal libnet version is now 1.1.6.
(#3528)
configure: added new --enable-manpages-install option along with the
existing --enable-manpages. The new option would install pre-existing
manpages even without the DocBook tools installed.
(#3493)

Notes to developers

apphook: the concept of hook run modes were introduced, adding support for
two modes: AHM_RUN_ONCE (the original behavior) and AHM_RUN_REPEAT (the new
behavior with the hook repeatedly called after registration).
(#3561)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

0140454, Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler,
egorbeliy, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady,
Michael Ducharme, Norbert Takacs, Peter Czanik, Peter Kokai, Pratik raj,
Ryan Faircloth, Zoltan Pallagi

Assets 3

19 Nov 14:17

github-actions

syslog-ng-3.30.1

cbc4dfe

syslog-ng-3.30.1

3.30.1

Highlights

filter template function
support proxy-protocol

Features

kafka (C implementation):
- Added template support to topic().
- Added fallback-topic() option, which will be used, if the templated topic() yields an invalid topic name.
  (#3372)

transport: add proxy-protocol support

http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Example config, click to expand!

@version: 3.29

source s_tcp_pp {
    network(
        port(7777)
#        transport("proxied-tcp")
        transport("proxied-tls")
        tls(
             key-file("/openssl/certs/certs/server/server.rsa")
             cert-file("openssl/certs/certs/server/server.crt")
             ca-dir("/openssl/certs/certs/CA")
#             peer-verify("optional-untrusted")
             peer-verify("required-trusted")
         )
    );
};

destination d_file {
    file("/var/log/pp.log" template("$(format-json --scope nv-pairs)\n"));
};

log {
    source(s_tcp_pp);
    destination(d_file);
};

([#3437](https://github.com//pull/3437))

filter: new template function

The new introduced filter template function will allow filtering lists based on a filter expression.

For example this snippet removes odd numbers

log {
  source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); };
  destination {
     file("/dev/stdout"
           template("$(filter ('$(% $_ 2)' eq '0') $INPUT)\n)")
     );
  };
};

(#3426)

file, network, program destinations: : new truncate_size option introduced to truncate an output message to a specified max size. default value is -1 (disabled).
```
network("127.0.0.1" truncate_size(100));
```
new stats counters:
```
dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_count;1
dst.network;d_local#0;udp,127.0.0.1:1111;a;truncated_bytes;1
```
(#3474)
network: add FreeBSD support for the so_reuseport(yes) the same as in linux SO_REUSEPORT (FreeBSD uses SO_REUSEPORT_LB flag).
(#3438)
date-parser: %z accepts local timezone std format as well
(#3453)
syslog-format: accepting longer sdata keys

Triggered by #3197
At the end of the discussion in the mentioned issue, we decided
to change the parser and accept longer than 32 character ID's.
(#3244)
systemd-journal: add namespace() option
This option accepts a string which is identical to the --namespace option of journalctl.
For systems defining this option with a systemd version older than v245 a warning is issued.
(#3358)

Bugfixes

date-parse: %Z should parse the same timezones as %z not just local and gmt
(#3453)
python: printing the exception instead of None (if compiled with clang)
(#3405)
network/udp: message was lost (not sent) if it was too large, and a time reopen amount of time needed to expire to send the next message lowering the thoughtput. now it is truncated at 65507.
(#3474)
tlscontext: support IPv6 X509v3 Subject Alternative Name

Fixes #3465
(#3466)

map: pass $_ to if correctly.

Prior this patchset, if did not receive $_ correctly.

After this change, these configurations will work:

log {
  source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); };
  destination {
     file("/dev/stdout"
           template("$(map $(if ('$(% $_ 2)' eq '0') 'even' 'odd') $INPUT)'\n)")
     );
  };
};

(#3426)

systemd-journal: add namespace to the persist name
(#3407)
syslog-ng: fixed numerous spelling mistakes in messages generated by syslog-ng
(#3398)
network: fix TLS certificate hostname verification when using failover() servers

For TLS certificate hostname verification, the certificate's hostname needs to be compared to the configured hostname
of the primary and each failover server. syslog-ng used always the primary server's name incorrectly.
(#3447)
afsocket: syslog-ng fails to bind() after config revert

When having a program source or destination and a network destination in the
config, if we reload with an invalid config, syslog-ng crashes, as it cannot init
the old network source, because its address is in use.
(#3416)
syslog-ng-ctl: when syslog-ng gets stuck on executing a heavy stats-ctl command, should be
able to do a graceful shutdown when it is requested.
(#3349)
json-parser: fix parsing 64 bit numbers (currently 32 bit was a limit)
(#3403)
usertty(): on each tty open error an error mesage and a 10 minutes long disabling of the usertty() destination has been added.
Until now, the usertty() destination were only disabled for blocking write() calls.
(#3473)

Notes to developers

Proxy protocol support added to loggen.

Four new options added to loggen to suppport the proxy protocol:
- --proxied : Generate PROXY protocol v1 header
- --proxy-src-ip : Set the source IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X).
- --proxy-dst-ip : Set the destination IP for the PROXY protocol v1 header. If not specified a random IP address generated (192.168.1.X).
- --proxy-src-port : Set the source port for the PROXY protocol v1 header. If not specified a random port generated in the range 5000-10000.
- --proxy-dst-port : Set the destination port for the PROXY protocol v1 header. If not specified the port number 514 will be used.
  (#3462)
bison: Minimum required version is now 3.4.2.
You still only need bison, if you are building from git source or changing the grammar in the released source tarball.
(#2526)
Template evaluation related function signatures changed.

A new structure LogTemplateEvalOptions is introduced to group parameters together.
(#3426)

Other changes

json-parser: change every per message logs that was higher than debug to debug
(#3401)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler,
Boris Korzun, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady,
Norbert Takacs, Peter Kokai, Viktor Juhasz, Vivin Peris, Zoltan Pallagi,
bjoe2k4

Assets 3

28 Aug 14:57

lbudai

syslog-ng-3.29.1

d5900a2

syslog-ng-3.29.1

3.29.1

Highlights

panos-parser(): parse Palo Alto PAN-OS logs

Example:

@include "scl.conf"

log {
  source { network(transport("udp")); };

  parser { panos-parser(); };

  destination {
   elasticsearch-http(
     index("syslog-ng-${YEAR}-${MONTH}-${DAY}")
     type("")
     url("http://localhost:9200/_bulk")
     template("$(format-json
       --scope rfc5424
       --scope dot-nv-pairs --rekey .* --shift 1 --exclude *future_* --exclude *dg_hier_level_*
       --scope nv-pairs --exclude DATE --key ISODATE @timestamp=${ISODATE})")
   );
  };
};

(#3234)

Features

snmptrap: improve error message when missing dependency
(#3363)
disk queue: reduce memory usage during load
(#3352)
Allow dupnames flag to be used in PCRE expressions, allowing duplicate names for named subpatterns
as explained here: https://www.pcre.org/original/doc/html/pcrepattern.html#SEC16 .

Example:
```
filter f_filter1 {
  match("(?<FOOBAR>bar)|(?<FOOBAR>foo)" value(MSG) flags(store-matches, dupnames));
};
```
(#3381)

Bugfixes

filter/regex: if there was a named match (?..)? that is optional to match, the previose or the next named matches might not be saved as named match.
(#3393)
tls: Fixed a bug, where ecdh-curve-list() were not applied at client side.
(#3356)
scratch-buffers: fix global.scratch_buffers_bytes.queued counter bug
This bug only affected the stats_counter value, not the actual memory usage (i.e. memory usage was fine before)
(#3355)
wsl: fix infinite loop during startup
(#3340)
openbsd: showing grammar debug info for openbsd too, when -y command line option is used
(#3339)
stats-query: speedup syslog-ng-ctl query get "*" command.

An algorithmic error view made syslog-ng-ctl query get "*" very slow with large number of counters.
(#3376)
syslogformat: fixing crashing with small invalid formatted logs see example in #3328
(#3364)
cfg: fix config reload crash via introducing on_config_inited in LogPipe
(#3176)
config: fix error reporting
- Error reporting was fixed for lines longer than 1024 characters.
- The location of the error was incorrectly reported in some cases.
  (#3383)
disk queue: fix possible crash during load, and possible false positive corruption detection
(#3342)
db-parser, pdbtool, graphite-output: fix glib assertion error

The assertion happened in these cases
- dbparser database load
- argument parsing in graphite-output
- pdbtool merge commad
Syslog-ng emitted a glib assertion warning in the cases above, even in successful executions.

If G_DEBUG=fatal-warnings environment variable was used, the warning turned into a crash.
(#3344)
stats: fix stats-ctl query crash when trying to reset all the counters
syslog-ng-ctl query get '*' --reset
(#3361)

Packaging

RHEL 7 packaging: fix logrotate file conflict with rsyslog
(#3324)
Debian packaging: python3-nose was removed from package dependencies.
Pytest will run Python related unittests (for modules/python/pylib/syslogng/debuggercli/tests/)
instead of nose.
(#3343)

Notes to developers

light: test for assertion errors in glib for each testcases
(#3344)
Fix signal handling when an external library/plugin sets SIG_IGN

Previously, setting SIG_IGN in a plugin/library (for example, in a Python module) resulted in a crash.
(#3338)
func-test: removed logstore_reader check, which was never reached
(#3236)
plugin_skeleton_creator: fixing a compiler switch

Wrong compiler switch used in plugin_skeleton_creator. This caused a compiler warning. The grammar debug info did not appear for that module, when -y command line option was used.
(#3339)
Light test framework: get_stats and get_query functions to DestinationDriver class

Two new functions added to DestinationDriver class which can be used for getting the stats
and query output of syslog-ng-ctl.
(#3211)

Other changes

internal(): limit the size of internal()'s temporary queue

The internal() source uses a temporary queue to buffer messages.
From now on, the queue has a maximum capacity, the log-fifo-size() option
can be used to change the default limit (10000).

This change prevents consuming all the available memory in special rare cases.
(#3229)
network plugins: better timer defaults for TCP keepalive

From now on, syslog-ng uses the following defaults for TCP keepalive:
- tcp-keepalive-time(): 60
- tcp-keepalive-intvl(): 10
- tcp-keepalive-probes(): 6
Note: so-keepalive() is enabled by default.
(#3357)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Christian Tramnitz, chunmeng, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady, MileK, Norbert Takacs, Peter Czanik, Péter Kókai, Terez Nemes.

Assets 3

23 Jun 19:26

lbudai

syslog-ng-3.28.1

d4523a1

syslog-ng-3.28.1

3.28.1

Highlights

http: add support for proxy option

Example:

log {
   source { system(); };
   destination { http( url("SYSLOG_SERVER_IP:PORT") proxy("PROXY_IP:PORT") method("POST") ); };
};

(#3253)

Features

map: template function

This template function applies a function to all elements of a list. For example: $(map $(+ 1 $_) 0,1,2) => 1,2,3.
(#3301)
use-syslogng-pid(): new option to all sources

If set to yes, syslog-ng overwrites the message's ${PID} macro to its own PID.
(#3323)

Bugfixes

affile: eliminate infinite loop in case of a spurious file path

If the template evaluation of a log message will result to a spurious
path in the file destination, syslog-ng refuses to create that file.
However the problematic log message was left in the msg queue, so
syslog-ng was trying to create that file again in time-reopen periods.
From now on syslog-ng will handle "permanent" file errors, and drop
the relevant msg.
(#3230)
Fix minor memory leaks in error scenarios
(#3265)
crypto: fix hang on boot due to lack of entropy
(#3271)
Fix IPv4 UDP destinations on FreeBSD

UDP-based destinations crashed when receiving the first message on FreeBSD due
to a bug in destination IP extraction logic.
(#3278)
network sources: fix TLS connection closure

RFC 5425 specifies that once the transport receiver gets close_notify from the
transport sender, it MUST reply with a close_notify.

The close_notify alert is now sent back correctly in case of TLS network sources.
(#2811)
disk-buffer: fixes possible crash, or fetching wrong value for logmsg nvpair
(#3281)
packaging/debian: fix mod-rdkafka Debian packaging
(#3282)
kafka destination: destination halts if consumer is down, and kafka's queue is filled
(#3305)
file-source: Throw error, when follow-freq() is set with a negative float number.
(#3306)
stats-freq: with high stats-freq syslog-ng emits stats immediately causing high memory and CPU usage
(#3320)
secure-logging: bug fixes (#3284)
- template arguments are now consistently checked
- fixed errors when mac file not provided
- fixed abort when derived key not provided
- fixed crash with slogkey missing parameters
- fixed secure-logging on 32-bit architectures
- fixed CMake build

Other changes

dbld: Fedora 32 support (#3315)
dbld: Removed Ubuntu Eoan (#3313)
secure-logging: improvements (#3284)
- removed 1500 message length limitation
- slogimport has been renamed to slogencrypt
- $(slog) will not start anymore when key is not found
- internal messaging (warning, debug) improvements
- improved memory handling and error information display
- CMake build improvements
- switched to GLib command line argument parsing
- the output of slogkey -s is now parsable
- manpage improvements

Notes to developers

dbld: devshell is now upgraded to Ubuntu Focal
(#3277)
dbld/devshell: Multiple changes:
- Added snmptrapd package.
- Added support for both python2 and python3.
  (#3222)
threaded-source: fully support default-priority() and default-facility()
(#3304)
CMake: fix libcap detection
(#3294)
Fix atomic_gssize_set() warning with new glib versions
(#3286)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Airbus Commercial Aircraft, Andras Mitzki, Antal Nemes, Attila Szakacs,
Balazs Scheidler, Gabor Nagy, Laszlo Budai, Laszlo Szemere, László Várady,
Péter Kókai, Vatsal Sisodiya, Vivin Peris.

Assets 3

30 Apr 17:08

lbudai

syslog-ng-3.27.1

0db9435

syslog-ng-3.27.1

3.27.1

Highlights

DESTIP/DESTPORT/PROTO: new macros. (#2899)
set-facility(): add new rewrite operation to change the syslog facility associated with the message. (#3136)
network tls: Added ca-file() option. With this option the user can set a bundled CA-file to verify the peer. (#3145)
Forward integrity and confidentiality of logs (#3121): It is an experimental feature currently, we are still working on the final form with the authors.

Features

DESTIP/DESTPORT/PROTO: new macros

These new macros express the destination ip, destination port and used protocol on a source.

The use-case behind the PR is as follows:
- someone has an appliance which sends out log messages via both UDP and TCP
- the format of the two are different, and he wants to capture either with the simplest possible filter
- netmask() doesn't work because the IP addresses are the same
- host() doesn't work because the hostnames are the same
Example:
```
log {
  source { network(localip(10.12.15.215) port(5555) transport(udp)); };
  destination { file("/dev/stdout" template("destip=$DESTIP destport=$DESTPORT proto=$PROTO\n")); };
};
```
Output:
```
destip=10.12.15.215 destport=5555 proto=17
```
(#2899)

set-facility(): add new rewrite operation to change the syslog facility
associated with the message.

log {
    source { system(); };
    if (program("postfix")) {
      rewrite { set-facility("mail"); };
    };
    destination { file("/var/log/mail.log"); };
    flags(flow-control);
};

(#3136)

network tls: Added ca-file() option. With this option the user can set a bundled CA-file to verify the peer.
(#3145)

http: When a HTTP response is received, emit a signal with the HTTP response code.
(Later it can be extended to read the response and parse it in a slot...).

This PR also extends the Python HTTP header module with the possibility of writing custom HTTP response code handlers. When someone implements an auth header plugin in Python, it could be useful (for example invalidating a cache).

Example config, click to expand!


@version: 3.25

python {
from syslogng import Logger

logger = Logger()

class TestCounter():
    def __init__(self, options):
        self.header = options["header"]
        self.counter = int(options["counter"])
        logger.debug(f"TestCounter class instantiated; options={options}")

    def get_headers(self, body, headers):
        logger.debug(f"get_headers() called, received body={body}, headers={headers}")

        response = ["{}: {}".format(self.header, self.counter)]
        self.counter += 1
        return response

    def on_http_response_received(self, http_code):
        self.counter += http_code
        logger.debug("HTTP response code received: {}".format(http_code))

    def __del__(self):
        logger.debug("Deleting TestCounter class instance")
};

source s_network {
  network(port(5555));
};

destination d_http {
    http(
        python_http_header(
            class("TestCounter")
            options("header", "X-Test-Python-Counter")
            options("counter", 11)
            # this means that syslog-ng will trying to send the http request even when this module fails
            mark-errors-as-critical(no)
        )
        url("http://127.0.0.1:8888")
    );
};

log {
    source(s_network);
    destination(d_http);
    flags(flow-control);
};

(#3159)

java/python: add support for the "arrow" syntax.
```
options("key" => "value")
```
(#3161) (#3247)
python: persist support for python

This feature enables users to persist data between reloads or restarts. The intended usage is to support bookmarking and acknowledgement in the future. It is not suitable for local database use cases.
(#3171)
rewrite: Added conditional set-tag() option. With this option the user can put condition statement inside set-tag option.
```
rewrite { set-tag("tag" condition(match("test" value("MSG")))); };
```
(#3190)
scl: add sumologic destinations: sumologic-syslog() and sumologic-http()
(#3194)
iterate: new template function

The iterate template function generates a series from an initial number and a next function.

For example you can generate a sequence of nonnegative numbers with
```
source {
  example-msg-generator(
    num(3)
    template("$(iterate $(+ 1 $_) 0)")
  );
};
```
(#3205)
telegram: new max-size option

Telegram message will be truncated for max-size size. Telegram does not accept message larger than 4096 utf8 characters. The default value is 4096.
(#3206)

example-message-generator : add support for values(name1 => value1, name2 => value2,..) syntax.

Example

@version: 3.27
log {
  source { example-msg-generator(template("message parameter")
                                 num(10)
                                 values("PROGRAM" => "program-name"
                                        "current-second" => "$C_SEC"
                                ));
         };
  destination { file(/dev/stdout template("$(format-json --scope all-nv-pairs)\n")); };
};

(#3237)

example-msg-generator: support freq(0) for fast message generation

log {
   source { example-msg-generator(freq(0) num(100)); };
   destination { file("/dev/stdout"); };
};

(#3245)

Bugfixes

file: changed time-reap() timer's schedule to respect the documentation (expires after last message)
(#3133)
dbld: fix building problems
- fix rpm package build on centos-7
- fix devshell image build
- fix ubuntu-trusty image build
- fix deb package build on ubuntu-trusty
- fix rpm package build on fedora-30
  (#3143)
tls (network): Properly log an error message, when key-file() or cert-file() is missing.
(#3145)
loggen: fix crash with invalid parameterization
(#3146)
format-json: fix printing of embedded zeros

Prior to 2.64.1, g_utf8_get_char_validated() in glib falsely identified embedded zeros as valid utf8 characters. As a result, format json printed the embedded zeroes as \u0000 instead of \x00. This change fixes this problem.
(#3175)
configure: fix --with-net-snmp configure option
(#3180)
python: fix Py_None reference counting in logger methods (trace, debug, info, warning, error)
(#3187)
afmongodb: do not build module when ENABLE_MONGODB=OFF
(#3188)
telegram: automatically truncate messages larger than 4096 utf8 characters to avoid telegram destination to get stuck
(#3206)
compat/glib: fix recursive call issue on CentOS-6/RHEL-6/platforms
(#3212)
timeutils: fix crash in %f conversion when non-numeric character is in usec field (e.g. ".asd123")
(#3270)

Packaging

macOS: add example startup configuration.
(#3172)
rpm: fix --without maxminddb option

If maxminddb development package was installed on the build system: rpmbuild fails if --without maxminddb was used.
(#3208)

Notes to developers

light: Support to relocate reports dir other than current base dir

For example

python -m pytest -lvs functional_tests/source_drivers/file_source/test_acceptance.py --installdir=/install --reports /tmp/

(#3157)

CONTRIBUTING.md: contribution guide updated
(#3174)
libtest: Now we install config_parse_lib.h, fake-time.h, mock-cfg-parser.h and queue_utils_lib.h
which help unit testing outside of core.
(#3179)
tests: Wait until snmptrapd process able to write traps into output file
(#3185)
mongodb: Replaced th...

Assets 3

05 Mar 11:10

lbudai

syslog-ng-3.26.1

f9a3495

syslog-ng-3.26.1

3.26.1

Highlights

file source: Added a new option to multi-line file sources: multi-line-timeout()
After waiting multi-line-timeout() seconds without reading new data from the file, the last (potentially partial)
message will be flushed and sent through the pipeline as a LogMessage.
Since the multi-line file source detects the end of a message after finding the beginning of the subsequent message
(indented or no-garbage/suffix mode), this option can be used to flush the last multi-line message
in the file after a multi-line-timeout()-second timeout.
There is no default value, i.e. this timeout needs to be explicitly configured.
Example config:
```
file("/some/folder/events"
    multi-line-mode("prefix-garbage")
    multi-line-prefix('^EVENT: ')
    multi-line-timeout(10)
    flags("no-parse")
);
```
(#2963)

python-http-header: Added this new plugin, which makes it possible for users to implement HTTP header plugins in Python language.
It is built on top of signal-slot mechanism: currently HTTP module defines only one signal, that is signal_http_header_request and python-http-header plugin implements a python binding for this signal. This means that when the signal_http_header_request signal is emitted then the connected slot executes the Python code.
The Python interface is:

def get_headers(self, body, headers):

It should return string List. The headers that will be appended to the request's header.
When the plugin fails, http module won't try to send the http request without the header items by default.
If you want http module to trying to send the request without these headers, just disable mark-errors-as-critical() function.
Original code was written by Ferenc Sipos.

Example config, click to expand!

@version: 3.26
python {
from syslogng import Logger
logger = Logger()
class TestCounter():
    def __init__(self, options):
        self.header = options["header"]
        self.counter = int(options["counter"])
        logger.debug(f"TestCounter class instantiated; options={options}")
    def get_headers(self, body, headers):
        logger.debug(f"get_headers() called, received body={body}, headers={headers}")
       
        response = ["{}: {}".format(self.header, self.counter)]
        self.counter += 1
        return response
    def __del__(self):
        logger.debug("Deleting TestCounter class instance")
};
source s_network {
  network(port(5555));
};
destination d_http {
    http(
        python_http_header(
            class("TestCounter")
            options("header", "X-Test-Python-Counter")
            options("counter", 11)
            # this means that syslog-ng will trying to send the http request even when this module fails
            mark-errors-as-critical(no)
        )
        url("http://127.0.0.1:8888")
    );
};
log {
    source(s_network);
    destination(d_http);
    flags(flow-control);
};

(#3123)

azure-auth-header: Added this new plugin, which generates authorization header for applications connecting to Azure.
It can be used as a building block in higher level SCLs.
Implemented as a signal-slot plugin.

Example config, click to expand!

@version: 3.26
@include "scl.conf"
destination d_http {
  http(
    url("http://127.0.0.1:8888")
    method("PUT")
    user_agent("syslog-ng User Agent")
    body("${ISODATE} ${MESSAGE}")
    azure-auth-header(
      workspace-id("workspace-id")
      secret("aa1a")
      method("POST")
      path("/api/logs")
      content-type("application/json")
    )
  );
};
source s_gen {
  example-msg-generator(num(1) template("Test message\n"));
};
log {
  source(s_gen);
  destination(d_http);
};

(#3078)

python: From now on users can specify a persist name template from python code.
```
@staticmethod
def generate_persist_name(options):
    return options["file_name"]
```
- Usage of this function is necessary, when one python destination is used multiple times in one config.
- Persist name from config takes precedence over generate_persist_name.
- Persist name is exposed through self.persist_name. (#3016)

Features

set-severity(): Added this new rewrite rule for changing message severity.
It receives a template and sets message severity by evaluating the template.
Numerical and textual severity levels are both supported.
Examples:
```
rewrite {
  set-severity("info");
  set-severity("6");
  set-severity("${.json.severity}");
};
```
(#3115)
$(list-search): Added a new template function, which returns the first index of a pattern in a list.
Starts the search at start_index. 0 based. If not found, returns empty string.
Usage: $(list-search --mode MODE <pattern> ${list})
Where mode can be: literal (default), prefix, substring, glob, pcre.
Add --start-index <index> to change the start index. (#2955)
config version: Made the config version check of the configuration more liberal by accepting version numbers
that had no changes relative to the current version. This means that if you are running 3.26 and the last
semantic change in the configuration was 3.22, then anything between 3.22 and 3.26 (inclusive) is accepted
by syslog-ng without a warning at startup. (#3074)
$SEVERITY instead of $LEVEL: syslog-ng now follows the RFC3164 trend of using severity instead of level
to refer to the severity of the message that is used in the template language ($SEVERITY),
filter expressions (severity()) and so on. (#3128)
http: Added ssl_version("tlsv1_3") and ssl_version("no-tlsv13") options to respectively force and disable TLSv1.3. (#3063)
scl: Improved error message at init, when an scl is missing a dependency. (#3015)
geoip2: Added template() option as an alias for the positional argument string, to match the grammar convention. (#3051)
loggly: Added transport() option, so users can now use it with tls (or any network() supported transport). (#3149)
config-option-database: Added support for parser, diskq and hook-commands blocks. (#3029)

Bugfixes

configure.ac: Fixed gethostbyname() function location detection (#3135)
http: Fixed a crash, when workers() was set to 0. We do not allow nonnegative values anymore. (#3116)
snmp-dest: engine-id() option now handles 5 to 32 characters, instead of the strict 10 before. (#3058)
http: Fixed handling of ssl-version() option, which was ignored before.
Prior this fix, these values of ssl-version in http destination were ignored by syslog-ng:
tlsv1_0, tlsv1_1, tlsv1_2, tlsv1_3. (#3083)
network sources: Added workaround for a TLS 1.3 bug to prevent data loss.
Due to a bug in the OpenSSL TLS 1.3 implementation (openssl/openssl#10880),
it is possible to lose messages when one-way communication protocols are used, -
such as the syslog protocol over TLS (RFC 5425,
RFC 6587) - and the connection is closed by the client right after sending data.
The bug is in the TLS 1.3 session ticket handling logic of OpenSSL.

To prevent such data loss, we've disabled TLS 1.3 session tickets in all syslog-ng network sources.
Tickets are used for session resumption, which is currently not supported by syslog-ng.

The loggen testing tool also received some bugfixes (#3064), which reduce the
likelihood of data loss if the target of loggen has not turned off session tickets.

If you're sending logs to third-party OpenSSL-based TLS 1.3 collectors, we recommend turning session
tickets off in those applications as well until the OpenSSL bug is fixed. (#3082)
cmake: Now we install loggen headers, as we do with autotools. (#3067)
graylog2, format-gelf: Fixed sending empty message, when ${PID} is not set.
Also added a default value "-" to empty short_message and host as they are mandatory fields. (#3112)
loggen: fix dependency error with cmake + openssl from nonstandard location (#3062)
config-option-database: Fixed reading 'grammar' and 'parser' files on 'POSIX' environment (#3125)
file source: Fixed file source not able to process new message after log-msg-size() increase. (#3075)
checkpoint parser: Fixed parsing ISO timestamp. (#3056)
secret-storage: Fixed some cases, where diagnostical logs were truncated. (#3141)
loggen, dqtool: Fixed a crash, when writing error/debug message or relocating qfile. (#3069)
build: Fixed a compatibility related build error on Solaris 11. (#3070)
loggen: Fixed address resolution when only loopback interface was configured. (#3048)

Packaging

scl: Moved scl files to the core package. (#2979)
RHEL: Now we include the packaging/rhel/ folder in our release tarball. (#3071)
RHEL 8 / CentOS 8: Added RHEL 8 / CentOS 8 support to syslog-ng.spec (#3034)

Notes to developers

signal-slot-connector: Introduced a generic event handler interface for syslog-ng modules.
- The concept is simple:
  - There is a SignalSlotConnector which stores Signal - Slot connections
  - Signal : Slot = 1 : N, so multiple slots can be assigned to the same Signal.
  - When a Signal is emitted, the connected Slots are executed.
  - Signals are string lite...

Assets 3

12 Dec 12:31

lbudai

syslog-ng-3.25.1

b880087

syslog-ng-3.25.1

3.25.1

Highlights

http-destination: Users now can specify the action for any HTTP result code.
Use with response-action(response_code => action) in your http block.
Available actions are: success, retry, drop and disconnect. (#3007)
syslog-ng-cfg-db: Added a new script, which can provide the options of
sources and destinations queried by the user. This tool can make the configuration
of syslog-ng a lot easier. Use with ./syslog-ng-cfg-db.py from the
contrib/config_database dir.(#2997)
redis-destination: Improved the performance by 2 orders of magnitude.
In our labor environment, now it operates at 25k EPS. (#2972)

Features

create-dirs(): Added to pipe() source/destination, and standardize the behavior.
(#3018, #2635)
default-network-drivers: Added max-connections() option, to change the limit
from 10. (#2961)
checkpoint: Added support for timezone value at the end of timestamps. (#3033)
filter/rewrite: Added disable-jit flag to disable JIT PCRE compilation. (#2992, #2986)
syslog-ng-ctl: Added export-config-graph option to visualize config graph. (#2990)
build/travis: Added ARM64 arch support. (#2967)
build/dbld: Readded CentOS 6 support. (#2860, #2971, #3028)
python: Added Python 3.8 support. (#3017)

Bugfixes

tls: Fixed an infinite loop which occured, when a TLS connection broke. (#3026, #3009)
log-block: Fixed an issue, where inline network destinations disjointed
the rest of the config. (#2989, #2820)
kafka/network-load-balancer: Fixed a crash when an argument was set to empty. (#3002)
python-source: Fixed a memory corruption during reload. (#3014)
python-destination: Actually use return value of open method. (#2998, #2513)
python-fetcher: Fixed FETCH_NO_DATA and FETCH_TRY_AGAIN constants. (#3012)
python: Fixed python Exception reporting when no Exception happened. (#2995)
telegram: Fixed the syntax error of the use-system-cert-store() option. (#2977)
config: Throw error to single dots, which were ignored before. (#3000)
file-destination: Delay ACKs until messages are written to disk. This fixes message
drop on I/O error and message lost in the LogProtoFileWriter in case of a crash, by
retrying to send the message. (#2985)
http-destination: Handle global template options values. (#3020)
timeutils: Fixed month and day name parsing, when only the first 2 characters
matched. (#3035)
logmsg: Added default PRI value (LOG_USER | LOG_NOTICE) to log messages
created without initial parsing. (#2974)
packaging: Added ordering dependencies network.target and network-online.target
to the service files. (#2994, #2667)
amqp: Support older (0.7.1) version (#2999)
loggen: Set plugin path in installation time. (#3019)
timeutils/patterndb: Fixed some undefined behaviours. (#2969)
stomp: Fixed a buffer over-read on connection. (#2988)
pseudofile: Fixed a crash, when template() option is not set. (#2988)
wildcard-source: Fixed a crash, when max-files() was set to 0. (#2988)

Other changes

syslog-ng-debun: Various maintenance updates and small fixes. (#2993)
scl: Avoid @requires loading the plugins themselves. (#2887)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Clément Besnier,
Gabor Nagy, jadhavsumit98, Janos Szigetvari, Laszlo Budai, Laszlo Szemere,
László Várady, MikeLim, Nikita Uvarov, Norbert Takacs, pabloli, Péter Kókai,
Zoltan Pallagi.

Assets 3

Releases: syslog-ng/syslog-ng

syslog-ng-3.33.1

3.33.1

Highlights

Features

Bugfixes

Packaging

Other changes

Credits

syslog-ng-3.32.1

3.32.1

Highlights

Features

Bugfixes

Packaging

Credits

syslog-ng-3.31.2

3.31.2

Bugfixes

Packaging

Credits

syslog-ng-3.31.1

3.31.1

Highlights

Features

Bugfixes

Packaging

Notes to developers

Credits

syslog-ng-3.30.1

3.30.1

Highlights

Features

Bugfixes

Notes to developers

Other changes

Credits

syslog-ng-3.29.1

3.29.1

Highlights

Features

Bugfixes

Packaging

Notes to developers

Other changes

Credits

syslog-ng-3.28.1

3.28.1

Highlights

Features

Bugfixes

Other changes

Notes to developers

Credits

syslog-ng-3.27.1

3.27.1

Highlights

Features

Bugfixes

Packaging

Notes to developers

syslog-ng-3.26.1

3.26.1

Highlights

Features

Bugfixes

Packaging

Notes to developers

syslog-ng-3.25.1

3.25.1

Highlights

Features

Bugfixes

Other changes

Credits