Skip to content

Commit

Permalink
Remove GELF
Browse files Browse the repository at this point in the history 
Closes keycloak#27365

Signed-off-by: Václav Muzikář <[email protected]>
  • Loading branch information
@vmuzikar
vmuzikar committed Aug 19, 2024
1 parent 841e279 commit e3571e1
Show file tree
Hide file tree
Showing 26 changed files with 45 additions and 821 deletions.
6 changes: 6 additions & 0 deletions docs/documentation/release_notes/topics/26_0_0.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -129,3 +129,9 @@ The `_LEGACY` cookies also served another purpose, which was to allow login from
not recommended at all in production deployments of Keycloak, it is fairly frequent to access Keycloak over `http` outside
of `localhost`. As an alternative to the `_LEGACY` cookies Keycloak now doesn't set the `secure` flag and sets `SameSite=Lax`
instead of `SameSite=None` when it detects an insecure context is used.

= Removal of GELF logging handler

GELF support has been deprecated for a while now, and with this release it has been finally removed from {project_name}.
Other log handlers are available and fully supported to be used as a replacement of GELF, for example Syslog. For details
see the https://www.keycloak.org/server/logging[Logging guide].
6 changes: 6 additions & 0 deletions docs/documentation/upgrading/topics/changes/changes-26_0_0.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -147,3 +147,9 @@ If you wish to disable placeholder replacement for the `import` command, add the
You can still override automatic detection by specifying the `https-key-store-type` and `https-trust-store-type` explicitly. The same applies to the management interface and its `https-management-key-store-type`. Restrictions for the FIPS strict mode stay unchanged.

NOTE: The `+spi-truststore-file-*+` options and the truststore related options `+https-trust-store-*+` are deprecated, we strongly recommend to use System Truststore. For more details refer to the relevant https://www.keycloak.org/server/keycloak-truststore[guide].

= Removal of GELF logging handler

GELF support has been deprecated for a while now, and with this release it has been finally removed from {project_name}.
Other log handlers are available and fully supported to be used as a replacement of GELF, for example Syslog. For details
see the https://www.keycloak.org/server/logging[Logging guide].
332 changes: 1 addition & 331 deletions docs/guides/server/logging.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@ includedOptions="log log-*">
** console (_default_)
** file
** Syslog
<@profile.ifCommunity>
** GELF
</@profile.ifCommunity>

== Logging configuration
Logging is done on a per-category basis in {project_name}. You can configure logging for the root log level or for more specific categories such as `org.hibernate` or `org.keycloak`. This {section} describes how to configure logging.
Expand Down Expand Up @@ -75,7 +72,7 @@ To enable log handlers, enter the following command:

The available handlers are
<@profile.ifCommunity>
`console`, `file`, `syslog` and `gelf`.
`console`, `file`, `syslog`.
</@profile.ifCommunity>
<@profile.ifProduct>
`console`, `file` and `syslog`.
Expand Down Expand Up @@ -257,331 +254,4 @@ To use unstructured logging, enter the following command:

As you can see, the timestamp is present twice, so you can amend it correspondingly via the `--log-syslog-format` property.

<@profile.ifCommunity>

== Centralized logging using GELF
NOTE: The support for GELF log handler is deprecated and will be removed in a future {project_name} release.
The recommended alternative is using the Syslog log handler described above.

{project_name} can send logs to a centralized log management system such as the following:

* Graylog
* Logstash, inside the Elasticsearch, Logstash, Kibana (ELK) logging stack
* Fluentd, inside the Elasticsearch, Fluentd, Kibana (EFK) logging stack

{project_name} uses the https://quarkus.io/guides/centralized-log-management[Quarkus Logging GELF] extension to support these environments.

=== Enabling the GELF handler
To enable logging using GELF, add it to the list of activated log handlers.

.Example:
<@kc.start parameters="--log=\"console,gelf\""/>

=== Configuring the GELF handler

To configure the Host and Port of your centralized logging system, enter the following command and substitute the values with your specific values:

.Host and port of the GELF server:
<@kc.start parameters="--log=\"console,gelf\" --log-gelf-host=myhost --log-gelf-port=12345"/>

When the GELF handler is enabled, the host is using `localhost` as host value and UDP for communication. To use TCP instead of UDP, prefix the host value with `tcp:`. The Default port is `12201`.

.Include or exclude Stacktraces
{project_name} includes the complete Stacktrace inside the `StackTrace` field. To exclude this field, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-include-stack-trace=false"/>

.Configure the timestamp format
You can change the format of the `timestamp` field. For example, you can include the date and time down to seconds by entering the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-timestamp-format=\"\'yyyy-MM-dd HH:mm:ss\'\""/>

Alternatively, you could use the config file to avoid escaping:

[source, conf]
----
log-gelf-timestamp-format=yyyy-MM-dd HH:mm:ss
----

The default timestamp format is `yyyy-MM-dd HH:mm:ss,SSS`. You can use the https://docs.oracle.com/javase/10/docs/api/java/text/SimpleDateFormat.html[available SimpleDateFormat patterns] to define an appropriate timestamp.

.Configure the facility
The `facility` field is an indicator of the process or program that is the source of log messages. The default value is `keycloak`. To set this field to your preferred identifier, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-facility=MyKeycloak"/>

To use the CLI to configure {project_name} and use whitespaces for `facility`, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-facility=\"\'my keycloak\'\""/>

Alternatively, you could use the config file to avoid escaping:

[source, conf]
----
log-gelf-facility=my keycloak
----

.Configure the default message size
To change the default message size of 8kb (8192 bytes) of GELF log messages for {project_name}, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-max-message-size=16384"/>

The maximum size of one GELF log message is set in Bytes. The preceding example increases the size to 16kb. When messages exceed the maximum size, GELF submits the message in multiple chunks.

.Configure sending of message parameters
{project_name} includes message parameters of the occurred log event. These fields appear in the output as `MessageParam0`, `MessageParam1`, and so on, depending on the parameter length.
To switch off this behavior, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-include-message-parameters=false"/>

.Configure sending of source code location
{project_name} includes the `SourceClassName`, `SourceMethodName` and `SourceSimpleClassName` fields in the GELF log messages. These fields provide detail on the location of an exception that occurred. To stop sending these fields, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-include-location=false"/>

=== Example: Send logs to Graylog
The following example shows how to send {project_name} logs to the Graylog centralized logging stack. This example assumes you have a container tool such as https://www.docker.com/[docker] installed to start the `compose.yml`.

==== Starting the Graylog stack
The composed stack consists of:

* Graylog
* ElasticSearch
* MongoDB

[source, yaml]
----
version: '3.8'
services:
elasticsearch:
image: docker.io/elastic/elasticsearch:7.10.2
ports:
- "9200:9200"
environment:
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
discovery.type: "single-node"
networks:
- graylog
mongo:
image: mongo:4.4
networks:
- graylog
graylog:
image: graylog/graylog:4.3.3
ports:
- "9000:9000"
- "12201:12201/udp"
- "1514:1514"
environment:
GRAYLOG_HTTP_EXTERNAL_URI: "http://127.0.0.1:9000/"
# CHANGE ME (must be at least 16 characters)!
GRAYLOG_PASSWORD_SECRET: "forpasswordencryption"
# Password: admin
GRAYLOG_ROOT_PASSWORD_SHA2: "8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918"
networks:
- graylog
depends_on:
- elasticsearch
- mongo
networks:
graylog:
driver: bridge
----

Copy and save the example locally into a `compose.yml` file and enter this command:

[source,bash]
----
docker compose up -d
----
After a few seconds, the Stack is ready to serve requests.

==== Creating a Graylog UDP Input
Once the stack is running, you need to create a UDP Input Graylog listens to. You can create it from the Graylog web UI (System → Input → Select GELF UDP) available at http://localhost:9000 or using the API:

This `curl` example creates a new GELF UDP Input using the API and the default Graylog login credentials (admin/admin).

[source, bash]
----
curl -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "X-Requested-By: curl" -X POST -v -d \
'{"title":"udp input","configuration":{"recv_buffer_size":262144,"bind_address":"0.0.0.0","port":12201,"decompress_size_limit":8388608},"type":"org.graylog2.inputs.gelf.udp.GELFUDPInput","global":true}' \
http://localhost:9000/api/system/inputs
----

If the stack is still in the bootstrap phase, you receive a response containing `* Empty reply from server`. A successful response includes `HTTP/1.1 201 Created` to indicate that the UDP input is created.

==== Configure {project_name} to send logs using GELF
{project_name} needs to be configured to send logs using GELF. The appropriate configuration can be seen in the following keycloak.conf example. The example includes the `log-gelf-host` and `log-gelf-port` values. These are optional values that are included for illustration purposes; default values exist.

.{project_name} GELF Configuration

[source, conf]
----
log=console,gelf
log-gelf-host=localhost
log-gelf-port=12201
----

==== Graylog: See the results
. Open your web browser, go to `http://localhost:9000`.
. Log in to the Graylog web UI using the administrator credentials (admin/admin).
. Go to Streams, All Messages.
. Start updating the stream by pressing the Play button in the upper right corner.
. Start {project_name} using `start` or `start-dev` and your GELF config.

After a few seconds, {project_name} messages appear in the Graylog dashboard.

=== Example Setup using the ELK Stack
The following example shows how to send {project_name} logs to the ELK centralized logging stack. It assumes you have a container tool such as https://www.docker.com/[docker] installed to start the `compose.yml`.

==== Enable the logstash GELF plugin and create a pipeline
Logstash uses an input plugin that understands and parses the GELF format. To activate this plugin when you are starting the ELK stack later on, create a directory `pipelines` and a file `gelf.conf` located in this directory. Then create an empty `compose.yml` in the parent directory.

.File Structure:
[source]
----
/ELK
- compose.yml
- pipelines/
- gelf.conf
----


Add the following contents to `pipelines/gelf.conf` and save it:

[source, conf]
----
input {
gelf {
port => 12201
}
}
output {
stdout {}
elasticsearch {
hosts => ["http://elasticsearch:9200"]
}
}
----

This file activates and configures the logstash GELF plugin and points it to the right elasticsearch instance.

==== Starting the ELK stack
The composed stack consists of:

* ElasticSearch
* Logstash
* Kibana

Copy the following content to your `compose.yml` file:

[source, yaml]
----
# Launch Elasticsearch
version: '3.8'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
networks:
- elk
logstash:
image: docker.elastic.co/logstash/logstash-oss:6.8.2
volumes:
- source: ./pipelines #the source dir gelf.conf resides
target: /usr/share/logstash/pipeline
type: bind
ports:
- "12201:12201/udp"
- "5000:5000"
- "9600:9600"
networks:
- elk
depends_on:
- elasticsearch
kibana:
image: docker.elastic.co/kibana/kibana-oss:6.8.2
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
networks:
elk:
driver: bridge
----
Start the stack by entering the following command:

[source, bash]
----
docker compose up -d
----
After a few seconds the Stack should be ready to serve requests.

==== Configuring {project_name} to send logs using GELF
{project_name} needs to be configured to send logs using GELF. The appropriate configuration can be seen in the following keycloak.conf example. This example includes the `log-gelf-host` and `log-gelf-port` values. These are optional values, which are included for illustration purposes; default values exist.

.{project_name} Gelf Configuration

[source, conf]
----
log=console,gelf
log-gelf-host=localhost
log-gelf-port=12201
----

With this configuration applied, start {project_name} using `start-dev` or `start`.

==== Kibana: See the results
Open http://localhost:5601 to reach the Kibana dashboard. The exact configuration of a good monitoring dashboard is out of scope for this {section}. To find out if logs sent by {project_name} are delivered to Kibana, open the http://localhost:5601/app/kibana#/dev_tools/console?_g=()[Dev Tools] and execute the default `match_all` query. The logs should appear in the result field.

=== Configure a different log level for the GELF logger
To keep log storage costs and verbosity low, it is often wanted to only store a subset of the verbose application logs inside a centralized log management system. To configure {project_name} to use a different log level for the logs you want to ingest, use the following configuration:

[source, conf]
----
log=console,gelf
log-gelf-level=<desired-log-level>
...
----

.Example
To only see occurred log levels of warn and above in your centralized logging stack, but still see INFO level logs on the applications console logs, use the following configuration:

[source, conf]
----
log=console,gelf
log-level=INFO
log-gelf-level=warn
...
----

Looking at your ingested logs, you will see only messages of level warn or above will appear.

Keep in mind that `--log-level` is setting the leading log level, so for example when you invoke the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-level=error --log-gelf-level=info"/>

nothing below the error level will be sent to your logging stack. That means that even GELF in this example will receive only error level log messages.

=== Configure additional key values
Currently, the {project_name} configuration does not support partly dynamic configuration keys, as they are used in quarkus properties. For example, they are used when defining `quarkus.log.handler.gelf.additional-field.<my-name>.value`.

To add user-defined fields, you can provide these fields through a quarkus.properties file. See <@links.server id="configuration"/> and the _Using raw Quarkus properties_ section.

</@profile.ifCommunity>

</@tmpl.guide>
Loading

0 comments on commit e3571e1

Please sign in to comment.