Skip to content

Commit

Permalink
Merge pull request #954 from mmienko/support-key-value-fluent-api
Browse files Browse the repository at this point in the history
Support key values from slf4j's fluent api

Fixes #946
  • Loading branch information
philsttr authored Jun 17, 2023
2 parents af9f826 + 07314e5 commit 16c92fc
Show file tree
Hide file tree
Showing 11 changed files with 659 additions and 8 deletions.
71 changes: 66 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ The structure of the output, and the data it contains, is fully configurable.
* [LoggingEvent Fields](#loggingevent-fields)
* [Standard Fields](#standard-fields)
* [MDC fields](#mdc-fields)
* [Key Value Pair fields](#key-value-pair-fields)
* [Context fields](#context-fields)
* [Caller Info Fields](#caller-info-fields)
* [Custom Fields](#custom-fields)
Expand Down Expand Up @@ -990,11 +991,12 @@ The field names can be customized (see [Customizing Standard Field Names](#custo

### MDC fields

By default, each entry in the Mapped Diagnostic Context (MDC) (`org.slf4j.MDC`)
will appear as a field in the LoggingEvent.
By default, `LogstashEncoder`/`LogstashLayout` will write each
[Mapped Diagnostic Context (MDC) (`org.slf4j.MDC`)](https://www.slf4j.org/api/org/slf4j/MDC.html)
entry to the output.

This can be fully disabled by specifying `<includeMdc>false</includeMdc>`,
in the encoder/layout/appender configuration.
To disable writing MDC entries, add `<includeMdc>false</includeMdc>`
to the `LogstashEncoder`/`LogstashLayout` configuration.

You can also configure specific entries in the MDC to be included or excluded as follows:

Expand All @@ -1020,14 +1022,55 @@ It is a configuration error to specify both included and excluded key names.

By default, the MDC key is used as the field name in the output.
To use an alternative field name in the output for an MDC entry,
specify`<mdcKeyFieldName>mdcKeyName=fieldName</mdcKeyFieldName>`:
specify `<mdcKeyFieldName>mdcKeyName=fieldName</mdcKeyFieldName>`:

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<mdcKeyFieldName>key1=alternateFieldNameForKey1</mdcKeyFieldName>
</encoder>
```

### Key Value Pair Fields

Slf4j 2's [fluent API](https://www.slf4j.org/manual.html#fluent) supports attaching key value pairs to the log event.

`LogstashEncoder`/`LogstashLayout` will write each key value pair as a field in the output by default.

To disable writing key value pairs, add `<includeKeyValuePairs>false</includeKeyValuePairs>`
to the `LogstashEncoder`/`LogstashLayout` configuration.

You can also configure specific key value pairs to be included or excluded as follows:

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<includeKeyValueKeyName>key1ToInclude</includeKeyValueKeyName>
<includeKeyValueKeyName>key2ToInclude</includeKeyValueKeyName>
</encoder>
```

or

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<excludeKeyValueKeyName>key1ToExclude</excludeKeyValueKeyName>
<excludeKeyValueKeyName>key2ToExclude</excludeKeyValueKeyName>
</encoder>
```

When key names are specified for inclusion, then all other keys will be excluded.
When key names are specified for exclusion, then all other keys will be included.
It is a configuration error to specify both included and excluded key names.

By default, the key is used as the field name in the output.
To use an alternative field name in the output for an key value pair,
specify`<keyValuePairsKeyFieldName>keyName=fieldName</keyValuePairsKeyFieldName>`:

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<keyValueKeyFieldName>key1=alternateFieldNameForKey1</keyValueKeyFieldName>
</encoder>
```

### Context fields

By default, each property of Logback's Context (`ch.qos.logback.core.Context`)
Expand Down Expand Up @@ -2593,6 +2636,24 @@ The provider name is the xml element name to use when configuring. Each provider
</ul>
</td>
</tr>
<tr>
<td valign="top"><tt>keyValuePairs</tt></td>
<td>
<p>Outputs key value pairs added via slf4j's fluent api.
Will include all key value pairs by default.
When key names are specified for inclusion, then all other keys will be excluded.
When key names are specified for exclusion, then all other keys will be included.
It is a configuration error to specify both included and excluded key names.
</p>
<ul>
<li><tt>fieldName</tt> - Sub-object field name (no sub-object)</li>
<li><tt>includeKeyName</tt> - Name of keys to include (all)</li>
<li><tt>excludeKeyName</tt> - Name of keys to exclude (none)</li>
<li><tt>keyFieldName</tt> - Strings in the form <tt>keyName=fieldName</tt>
that specify an alternate field name to output for specific key (none)</li>
</ul>
</td>
</tr>
<tr>
<td valign="top"><tt>message</tt></td>
<td><p>Formatted log event message.</p>
Expand Down
65 changes: 65 additions & 0 deletions src/main/java/net/logstash/logback/LogstashFormatter.java
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import net.logstash.logback.composite.LogstashVersionJsonProvider;
import net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider;
import net.logstash.logback.composite.loggingevent.CallerDataJsonProvider;
import net.logstash.logback.composite.loggingevent.KeyValuePairsJsonProvider;
import net.logstash.logback.composite.loggingevent.LogLevelJsonProvider;
import net.logstash.logback.composite.loggingevent.LogLevelValueJsonProvider;
import net.logstash.logback.composite.loggingevent.LoggerNameJsonProvider;
Expand All @@ -45,6 +46,7 @@
import ch.qos.logback.core.spi.ContextAware;
import com.fasterxml.jackson.databind.JsonNode;
import org.slf4j.MDC;
import org.slf4j.event.KeyValuePair;

/**
* A {@link LoggingEventCompositeJsonFormatter} that contains a common
Expand Down Expand Up @@ -87,6 +89,11 @@ public class LogstashFormatter extends LoggingEventCompositeJsonFormatter {
* to the logic in {@link MdcJsonProvider}.
*/
private MdcJsonProvider mdcProvider = new MdcJsonProvider();
/**
* When not null, {@link KeyValuePair} properties will be included according
* to the logic in {@link KeyValuePairsJsonProvider}.
*/
private KeyValuePairsJsonProvider keyValuePairsProvider = new KeyValuePairsJsonProvider();
private GlobalCustomFieldsJsonProvider<ILoggingEvent> globalCustomFieldsProvider;
/**
* When not null, markers will be included according
Expand Down Expand Up @@ -123,6 +130,7 @@ public LogstashFormatter(ContextAware declaredOrigin, boolean includeCallerData,
getProviders().addStackTrace(this.stackTraceProvider);
getProviders().addContext(this.contextProvider);
getProviders().addMdc(this.mdcProvider);
getProviders().addKeyValuePairs(this.keyValuePairsProvider);
getProviders().addGlobalCustomFields(this.globalCustomFieldsProvider);
getProviders().addTags(this.tagsProvider);
getProviders().addLogstashMarkers(this.logstashMarkersProvider);
Expand Down Expand Up @@ -222,6 +230,22 @@ public void setIncludeMdc(boolean includeMdc) {
}
}

public boolean isIncludeKeyValuePairs() {
return this.keyValuePairsProvider != null;
}

public void setIncludeKeyValuePairs(boolean includeKeyValuePairs) {
if (isIncludeKeyValuePairs() != includeKeyValuePairs) {
if (includeKeyValuePairs) {
keyValuePairsProvider = new KeyValuePairsJsonProvider();
addProvider(keyValuePairsProvider);
} else {
getProviders().removeProvider(keyValuePairsProvider);
keyValuePairsProvider = null;
}
}
}

public boolean isIncludeTags() {
return this.tagsProvider != null;
}
Expand Down Expand Up @@ -299,6 +323,44 @@ public void addMdcKeyFieldName(String mdcKeyFieldName) {
}
}

public List<String> getIncludeKeyValueKeyNames() {
return isIncludeKeyValuePairs()
? keyValuePairsProvider.getIncludeKeyNames()
: Collections.emptyList();
}

public void addIncludeKeyValueKeyName(String includedKeyValueKeyName) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.addIncludeKeyName(includedKeyValueKeyName);
}
}
public void setIncludeKeyValueKeyNames(List<String> includeKeyValueKeyNames) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.setIncludeKeyNames(includeKeyValueKeyNames);
}
}

public List<String> getExcludeKeyValueKeyNames() {
return isIncludeKeyValuePairs()
? keyValuePairsProvider.getExcludeKeyNames()
: Collections.emptyList();
}
public void addExcludeKeyValueKeyName(String excludedKeyValueKeyName) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.addExcludeKeyName(excludedKeyValueKeyName);
}
}
public void setExcludeKeyValueKeyNames(List<String> excludeKeyValueKeyNames) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.setExcludeKeyNames(excludeKeyValueKeyNames);
}
}
public void addKeyValueKeyFieldName(String keyValueKeyFieldName) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.addKeyFieldName(keyValueKeyFieldName);
}
}

public boolean isIncludeContext() {
return contextProvider != null;
}
Expand Down Expand Up @@ -388,6 +450,9 @@ public void addProvider(JsonProvider<ILoggingEvent> provider) {
} else if (provider instanceof MdcJsonProvider) {
getProviders().removeProvider(this.mdcProvider);
this.mdcProvider = (MdcJsonProvider) provider;
} else if (provider instanceof KeyValuePairsJsonProvider) {
getProviders().removeProvider(this.keyValuePairsProvider);
this.keyValuePairsProvider = (KeyValuePairsJsonProvider) provider;
}
getProviders().addProvider(provider);
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
/*
* Copyright 2013-2022 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package net.logstash.logback.composite.loggingevent;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import net.logstash.logback.composite.AbstractFieldJsonProvider;
import net.logstash.logback.composite.FieldNamesAware;
import net.logstash.logback.fieldnames.LogstashFieldNames;

import ch.qos.logback.classic.spi.ILoggingEvent;
import com.fasterxml.jackson.core.JsonGenerator;
import org.slf4j.event.KeyValuePair;

/**
* Includes key value pairs added from slf4j's fluent api in the output according to
* {@link #includeKeyNames} and {@link #excludeKeyNames}.
*
* <p>There are three valid combinations of {@link #includeKeyNames}
* and {@link #excludeKeyNames}:</p>
*
* <ol>
* <li>When {@link #includeKeyNames} and {@link #excludeKeyNames}
* are both empty, then all entries will be included.</li>
* <li>When {@link #includeKeyNames} is not empty and
* {@link #excludeKeyNames} is empty, then only those entries
* with key names in {@link #includeKeyNames} will be included.</li>
* <li>When {@link #includeKeyNames} is empty and
* {@link #excludeKeyNames} is not empty, then all entries except those
* with key names in {@link #excludeKeyNames} will be included.</li>
* </ol>
*
* <p>It is a configuration error for both {@link #includeKeyNames}
* and {@link #excludeKeyNames} to be not empty.</p>
*
* <p>By default, for each key value pair, the key is output as the field name.
* This can be changed by specifying an explicit field name to use for a ke
* via {@link #addKeyFieldName(String)}</p>
*
* <p>If the fieldName is set, then the pairs will be written
* to that field as a subobject.
* Otherwise, the pairs are written inline.</p>
*/
public class KeyValuePairsJsonProvider extends AbstractFieldJsonProvider<ILoggingEvent> implements FieldNamesAware<LogstashFieldNames> {

/**
* See {@link KeyValuePairsJsonProvider}.
*/
private List<String> includeKeyNames = new ArrayList<>();

/**
* See {@link KeyValuePairsJsonProvider}.
*/
private List<String> excludeKeyNames = new ArrayList<>();

private final Map<String, String> keyFieldNames = new HashMap<>();

@Override
public void start() {
if (!this.includeKeyNames.isEmpty() && !this.excludeKeyNames.isEmpty()) {
addError("Both includeKeyNames and excludeKeyNames are not empty. Only one is allowed to be not empty.");
}
super.start();
}

@Override
public void writeTo(JsonGenerator generator, ILoggingEvent event) throws IOException {
List<KeyValuePair> keyValuePairs = event.getKeyValuePairs();
if (keyValuePairs == null || keyValuePairs.isEmpty()) {
return;
}

String fieldName = getFieldName();
if (fieldName != null) {
generator.writeObjectFieldStart(getFieldName());
}

for (KeyValuePair keyValuePair : keyValuePairs) {
if (keyValuePair.key != null && keyValuePair.value != null
&& (includeKeyNames.isEmpty() || includeKeyNames.contains(keyValuePair.key))
&& (excludeKeyNames.isEmpty() || !excludeKeyNames.contains(keyValuePair.key))) {

String key = keyFieldNames.get(keyValuePair.key);
if (key == null) {
key = keyValuePair.key;
}
generator.writeFieldName(key);
generator.writeObject(keyValuePair.value);
}
}

if (fieldName != null) {
generator.writeEndObject();
}
}

@Override
public void setFieldNames(LogstashFieldNames fieldNames) {
setFieldName(fieldNames.getKeyValuePair());
}

public List<String> getIncludeKeyNames() {
return Collections.unmodifiableList(includeKeyNames);
}

public void addIncludeKeyName(String includedKeyName) {
this.includeKeyNames.add(includedKeyName);
}

public void setIncludeKeyNames(List<String> includeKeyNames) {
this.includeKeyNames = new ArrayList<>(includeKeyNames);
}

public List<String> getExcludeKeyNames() {
return Collections.unmodifiableList(excludeKeyNames);
}

public void addExcludeKeyName(String excludedKeyName) {
this.excludeKeyNames.add(excludedKeyName);
}

public void setExcludeKeyNames(List<String> excludeKeyNames) {
this.excludeKeyNames = new ArrayList<>(excludeKeyNames);
}

public Map<String, String> getKeyFieldNames() {
return keyFieldNames;
}

/**
* Adds the given keyFieldName entry in the form keyName=fieldName
* to use an alternative field name for an KeyValuePair key.
*
* @param keyFieldName a string in the form kvpKeyName=fieldName that identifies what field name to use for a specific KeyValuePair key.
*/
public void addKeyFieldName(String keyFieldName) {
String[] split = keyFieldName.split("=");
if (split.length != 2) {
throw new IllegalArgumentException("keyFieldName (" + keyFieldName + ") must be in the form keyName=fieldName");
}
keyFieldNames.put(split[0], split[1]);
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@ public void addContextName(ContextNameJsonProvider provider) {
public void addMdc(MdcJsonProvider provider) {
addProvider(provider);
}
public void addKeyValuePairs(KeyValuePairsJsonProvider provider) {
addProvider(provider);
}
public void addTags(TagsJsonProvider provider) {
addProvider(provider);
}
Expand Down
Loading

0 comments on commit 16c92fc

Please sign in to comment.