Skip to content
This repository has been archived by the owner on Jun 1, 2024. It is now read-only.

Commit

Permalink
Automatically handle TypeName parameter for different versions of E…
Browse files Browse the repository at this point in the history
…lasticsearch (<7, 7 or higher) (#462)

* fix file references in the Visual Studio Solution file.

* Do not set `TypeName` by default any more.
* last version of Elasticsearch that supported user-defined `_type` field - v6.8.x is running out of support on 2022-02-10 (https://www.elastic.co/support/eol)

* Automatically handle `ElasticsearchSinkOptions.TypeName` for different versions Elasticsearch (<7, 7+) when `ElasticsearchSinkOptions.DetectElasticsearchVersion` is enabled.

* Add unit test for automatic settings of `TypeName` when `DetectElasticsearchVersion` is set to `true`.
- Two methods used - instantiate `ElasticsearchSinkState` directly and via `LoggerConfiguration`

* Add Elasticsearch v8 template + parsing of Elasticsearch major version to `int` + decision branch for which version of index-template API to use + removal of obsolete `ElasticsearchTemplateProvider.GetTemplate(...)` method overload.

* Upgrade to .NET 6.0 and update test-frameworks related NuGet pacakges

* Upgrade to Elasticsearch.NET 7.17.5 + handle new "preflight" request

https://discuss.elastic.co/t/the-client-is-unable-to-verify-that-the-server-is-elasticsearch-due-to-an-unsuccessful-product-check-call-some-functionality-may-not-be-compatible-if-the-server-is-running-an-unsupported-product/310969/9

"The 7.16 client performs a pre-flight GET request to the root URL of the server before the first request.".

* Make `ConnectionStub` a bit more robust .

* Use `System.Version` to parse Elasticsearch server version number (similar to what `Elasticsearch.Net` does)

* Update NuGet packages

* Replace obsolete NuGet package `Serilog.Sinks.ColoredConsole` with `Serilog.Sinks.Console`

* Update `Serilog.Sinks.PeriodicBatching` package and reimplent `ElasticsearchSink` so that it does not use obsolete `PeriodicBatchingSink` constructor.

* Better handling of Elasticsearch server version number in mocked `ConnectionStub`

* Cleanup: refactor to use single `JsonEquals` method.

* Turn on `DetectElasticSearchVersion` option by default. Assume version 7 on fallback.

* Cleanup: remove unused namespaces

* Cleanup: move `ElasticsearchSinkTestsBase` into `Stubs` subfolder.

* Refactor: extract `ConnectionStub` into a separate file.

* Fix: json comparison in .NET Framework 4.6+

* Run unit-tests on multiple .NET frameworks.

* Cleanup: remove unused NUnit runner package.

* Use newer, built-in compilation constants.
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/preprocessor-directives#conditional-compilation

* Use standard MSBuild property `IsPackable` for clarity.

* Cleanup: remove unused package refrence.

* Update GitHub actions

* docs: updated documentation to reflect  changes in behavior of the sink.

Co-authored-by: Nenad Vicentic <[email protected]>
  • Loading branch information
nenadvicentic and Nenad Vicentic authored Jan 23, 2023
1 parent 306a3aa commit fa5b097
Show file tree
Hide file tree
Showing 61 changed files with 1,002 additions and 788 deletions.
20 changes: 20 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [9.0.0] - 2023-01-23

### Added
- PR #462

### Major Changes
- `DetectElasticsearchVersion` is set to `true` by default.
- When `DetectElasticsearchVersion` is set to `false` Elasticsearch version 7 is assumed (as it has broadest wire-compatibility at the moment - v7 and v8)
- When `DetectElasticsearchVersion` is set to `true`, `TypeName` is handled automatically across different versions of Elasticserach (6.x to 8.x). For example, user-defined name will NOT be used on v7 and v8. Also, correct templates endpoint will be picked up.
- Elasticsearch 8.x endpoint for templates is supported (`_index_template`)
- Internal class `ElasticsearchVersionManager` has been added, mainly to handle situations where detection of version fails or when it is disabled. In the case of fallback, sink will assume "default" version 7.
- Elasticsearch.NET client version 7.15.2 (latest version 7, until new `Elastic.Clients.Elasticsearch` 8.x catches up functional parity with 7.x)

### Other Changes
- Nuget pacakges have been updated (except for the Elasticsearch integration-tests related packages)
- Most of the `ElasticserachSink` functionality has been moved into internal `BatchedElasticsearchSink` class that inherits from `IBatchedLogEventSink`, so it complies with new recommended way of integration with `PeriodicBatchingSink` and we don't use obsolete constructors.
- `ConnectionStub` was moved out of `ElasticsearchSinkTestsBase` and extended. Both are now in `/Stubs` subfolder. Newer versions of Elasticsearch.NET client are now using "pre-flight" request to determine if endpoint is Elasticsearch and if it is indeed between 6.x and 8.x. `ConnectionStub` had to accommodate for that.
- Unit tests have been fixed/added accordingly, running on multiple target frameworks (`net6`, `net7` and `net48`).
- Built-in .NET SDK conditional compilation symbols are now used (e.g NETFRAMEWORK).

## [9.0.0] - 2022-04-29

### Fixed
Expand Down
70 changes: 68 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ The Serilog Elasticsearch sink project is a sink (basically a writer) for the Se
* Starting from version 3, compatible with Elasticsearch 2.
* Version 6.x supports the new Elasticsearch.net version 6.x library.
* From version 8.x there is support for Elasticsearch.net version 7.
* From version 9.x there is support for Elasticsearch.net version 8. Version detection is enabled by default, in which case `TypeName` is handled automatically across major versions 6, 7 and 8.


## Quick start
Expand All @@ -40,16 +41,80 @@ The Serilog Elasticsearch sink project is a sink (basically a writer) for the Se
Install-Package serilog.sinks.elasticsearch
```

Register the sink in code or using the appSettings reader (from v2.0.42+) as shown below. Make sure to specify the version of ES you are targeting. Be aware that the AutoRegisterTemplate option will not overwrite an existing template.
Simplest way to register this sink is to use default configuration:

```csharp
var loggerConfig = new LoggerConfiguration()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"));
```

Or, if using .NET Core and `Serilog.Settings.Configuration` Nuget package and `appsettings.json`, default configuration would look like this:

```json
{
"Serilog": {
"Using": [ "Serilog.Sinks.Elasticsearch" ],
"MinimumLevel": "Warning",
"WriteTo": [
{
"Name": "Elasticsearch",
"Args": {
"nodeUris": "http://localhost:9200"
}
}
],
"Enrich": [ "FromLogContext", "WithMachineName" ],
"Properties": {
"Application": "ImmoValuation.Swv - Web"
}
}
}
```

More elaborate configuration, using additional Nuget packages (e.g. `Serilog.Enrichers.Environment`) would look like:

```json
{
"Serilog": {
"Using": [ "Serilog.Sinks.Elasticsearch" ],
"MinimumLevel": "Warning",
"WriteTo": [
{
"Name": "Elasticsearch",
"Args": {
"nodeUris": "http://localhost:9200"
}
}
],
"Enrich": [ "FromLogContext", "WithMachineName" ],
"Properties": {
"Application": "My app"
}
}
}
```

This way the sink will detect version of Elasticsearch server (`DetectElasticsearchVersion` is set to `true` by default) and handle `TypeName` behavior correctly, based on the server version (6.x, 7.x or 8.x).

### Disable detection of Elasticsearch server version

Alternatively, `DetectElasticsearchVersion` can be set to `false` and certain option can be configured manually. In that case, the sink will assume version 7 of Elasticsearch, but options will be ignored due to a potential version incompatibility.

For example, you can configure the sink to force registeration of v6 index template. Be aware that the AutoRegisterTemplate option will not overwrite an existing template.

```csharp
var loggerConfig = new LoggerConfiguration()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200") ){
DetectElasticsearchVersion = false,
AutoRegisterTemplate = true,
AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv6
});
```

### Configurable properties

Besides a registration of the sink in the code, it is possible to register it using appSettings reader (from v2.0.42+) reader (from v2.0.42+) as shown below.

This example shows the options that are currently available when using the appSettings reader.

```xml
Expand All @@ -75,7 +140,8 @@ This example shows the options that are currently available when using the appSe
<add key="serilog:write-to:Elasticsearch.emitEventFailure" value="WriteToSelfLog" />
<add key="serilog:write-to:Elasticsearch.queueSizeLimit" value="100000" />
<add key="serilog:write-to:Elasticsearch.autoRegisterTemplate" value="true" />
<add key="serilog:write-to:Elasticsearch.autoRegisterTemplateVersion" value="ESv2" />
<add key="serilog:write-to:Elasticsearch.autoRegisterTemplateVersion" value="ESv7" />
<add key="serilog:write-to:Elasticsearch.detectElasticsearchVersion" value="false" /><!-- `true` is default -->
<add key="serilog:write-to:Elasticsearch.overwriteTemplate" value="false" />
<add key="serilog:write-to:Elasticsearch.registerTemplateFailure" value="IndexAnyway" />
<add key="serilog:write-to:Elasticsearch.deadLetterIndexName" value="deadletter-{0:yyyy.MM}" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.1</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
<IsPackable>false</IsPackable>
</PropertyGroup>

Expand All @@ -11,8 +11,8 @@
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="6.0.1" />
<PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="6.0.0" />
<PackageReference Include="Serilog" Version="2.11.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="4.0.1" />
<PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="4.1.0" />
</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Serilog" Version="2.11.0" />
<PackageReference Include="Serilog" Version="2.12.0" />
</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ public static LoggerConfiguration Elasticsearch(
/// <param name="failureSink">Sink to use when Elasticsearch is unable to accept the events. This is optionally and depends on the EmitEventFailure setting.</param>
/// <param name="singleEventSizePostingLimit"><see cref="ElasticsearchSinkOptions.SingleEventSizePostingLimit"/>The maximum length of an event allowed to be posted to Elasticsearch.default null</param>
/// <param name="templateCustomSettings">Add custom elasticsearch settings to the template</param>
/// <param name="detectElasticsearchVersion">Turns on detection of elasticsearch version via background HTTP call. This will also set `TypeName` automatically, according to the version of Elasticsearch.</param>
/// <param name="batchAction">Configures the OpType being used when inserting document in batch. Must be set to create for data streams.</param>
/// <returns>LoggerConfiguration object</returns>
/// <exception cref="ArgumentNullException"><paramref name="nodeUris"/> is <see langword="null" />.</exception>
Expand All @@ -151,7 +152,7 @@ public static LoggerConfiguration Elasticsearch(
string nodeUris,
string indexFormat = null,
string templateName = null,
string typeName = "logevent",
string typeName = null,
int batchPostingLimit = 50,
int period = 2,
bool inlineFields = false,
Expand All @@ -166,7 +167,7 @@ public static LoggerConfiguration Elasticsearch(
int queueSizeLimit = 100000,
string pipelineName = null,
bool autoRegisterTemplate = false,
AutoRegisterTemplateVersion autoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv2,
AutoRegisterTemplateVersion? autoRegisterTemplateVersion = null,
bool overwriteTemplate = false,
RegisterTemplateRecovery registerTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
string deadLetterIndexName = null,
Expand All @@ -182,7 +183,8 @@ public static LoggerConfiguration Elasticsearch(
long? singleEventSizePostingLimit = null,
int? bufferFileCountLimit = null,
Dictionary<string,string> templateCustomSettings = null,
ElasticOpType batchAction = ElasticOpType.Index)
ElasticOpType batchAction = ElasticOpType.Index,
bool detectElasticsearchVersion = true)
{
if (string.IsNullOrEmpty(nodeUris))
throw new ArgumentNullException(nameof(nodeUris), "No Elasticsearch node(s) specified.");
Expand All @@ -204,8 +206,6 @@ public static LoggerConfiguration Elasticsearch(
options.TemplateName = templateName;
}

options.TypeName = !string.IsNullOrWhiteSpace(typeName) ? typeName : null;

options.BatchPostingLimit = batchPostingLimit;
options.BatchAction = batchAction;
options.SingleEventSizePostingLimit = singleEventSizePostingLimit;
Expand Down Expand Up @@ -272,6 +272,8 @@ public static LoggerConfiguration Elasticsearch(

options.TemplateCustomSettings = templateCustomSettings;

options.DetectElasticsearchVersion = detectElasticsearchVersion;

return Elasticsearch(loggerSinkConfiguration, options);
}
}
Expand Down
2 changes: 1 addition & 1 deletion src/Serilog.Sinks.Elasticsearch/Properties/AssemblyInfo.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[assembly: AssemblyDescription("Serilog sink for Elasticsearch")]
[assembly: AssemblyCopyright("Copyright © Serilog Contributors 2018")]

[assembly: InternalsVisibleTo("Serilog.Tests, PublicKey=0024000004800000940000000602000000240000525341310004000001000100fb8d13fd344a1c" +
[assembly: InternalsVisibleTo("Serilog.Sinks.Elasticsearch.Tests, PublicKey=0024000004800000940000000602000000240000525341310004000001000100fb8d13fd344a1c" +
"6fe0fe83ef33c1080bf30690765bc6eb0df26ebfdf8f21670c64265b30db09f73a0dea5b3db4c9" +
"d18dbf6d5a25af5ce9016f281014d79dc3b4201ac646c451830fc7e61a2dfd633d34c39f87b818" +
"94191652df5ac63cc40c77f3542f702bda692e6e8a9158353df189007a49da0f3cfd55eb250066" +
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
<Authors>Michiel van Oudheusden, Martijn Laarman, Mogens Heller Grabe, Serilog Contributors</Authors>
<TargetFrameworks>netstandard2.0</TargetFrameworks>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<LangVersion>latest</LangVersion>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<AssemblyName>Serilog.Sinks.Elasticsearch</AssemblyName>
<AssemblyOriginatorKeyFile>../../assets/Serilog.snk</AssemblyOriginatorKeyFile>
Expand Down Expand Up @@ -35,11 +36,11 @@

<ProjectReference Include="..\Serilog.Formatting.Elasticsearch\Serilog.Formatting.Elasticsearch.csproj" />

<PackageReference Include="Serilog" Version="2.11.0" />
<PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Formatting.Compact" Version="1.1.0" />
<PackageReference Include="Serilog.Sinks.File" Version="5.0.0" />
<PackageReference Include="Serilog.Sinks.PeriodicBatching" Version="2.1.1" />
<PackageReference Include="Elasticsearch.Net" Version="7.8.1" />
<PackageReference Include="Serilog.Sinks.PeriodicBatching" Version="3.1.0" />
<PackageReference Include="Elasticsearch.Net" Version="7.17.5" />
<PackageReference Include="System.Diagnostics.DiagnosticSource" Version="6.0.0" />
</ItemGroup>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ private InvalidResult GetInvalidPayloadAsync(DynamicResponse baseResult, List<st
bool hasErrors = false;
foreach (dynamic item in items)
{
var itemIndex = item?[ElasticsearchSink.BulkAction(_elasticOpType)];
var itemIndex = item?[BatchedElasticsearchSink.BulkAction(_elasticOpType)];
long? status = itemIndex?["status"];
i++;
if (!status.HasValue || status < 300)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ protected override List<string> FinishPayLoad()
protected override void AddToPayLoad(string nextLine)
{
var indexName = _getIndexForEvent(nextLine, _date);
var action = ElasticsearchSink.CreateElasticAction(
var action = BatchedElasticsearchSink.CreateElasticAction(
opType: _elasticOpType,
indexName: indexName, pipelineName: _pipelineName,
id: _count + "_" + Guid.NewGuid(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,26 +25,39 @@

namespace Serilog.Sinks.Elasticsearch
{
public sealed class ElasticsearchSink : PeriodicBatchingSink
{
public ElasticsearchSink(ElasticsearchSinkOptions options)
: this(
new BatchedElasticsearchSink(options),
new PeriodicBatchingSinkOptions
{
BatchSizeLimit = options.BatchPostingLimit,
Period = options.Period,
EagerlyEmitFirstEvent = true,
QueueLimit = (options.QueueSizeLimit == -1) ? null : new int?(options.QueueSizeLimit)
}
)
{
}

private ElasticsearchSink(IBatchedLogEventSink batchedSink, PeriodicBatchingSinkOptions options)
: base(batchedSink, options)
{
}
}
/// <summary>
/// Writes log events as documents to ElasticSearch.
/// </summary>
public class ElasticsearchSink : PeriodicBatchingSink
internal sealed class BatchedElasticsearchSink : IBatchedLogEventSink
{

private readonly ElasticsearchSinkState _state;

/// <summary>
/// Creates a new ElasticsearchSink instance with the provided options
/// </summary>
/// <param name="options">Options configuring how the sink behaves, may NOT be null</param>
public ElasticsearchSink(ElasticsearchSinkOptions options)
: base(options.BatchPostingLimit, options.Period, options.QueueSizeLimit)
public BatchedElasticsearchSink(ElasticsearchSinkOptions options)
{
_state = ElasticsearchSinkState.Create(options);
_state.DiscoverClusterVersion();
_state.RegisterTemplateIfNeeded();
}

/// <summary>
/// Emit a batch of log events, running to completion synchronously.
/// </summary>
Expand All @@ -54,7 +67,7 @@ public ElasticsearchSink(ElasticsearchSinkOptions options)
/// or <see cref="M:Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.EmitBatchAsync(System.Collections.Generic.IEnumerable{Serilog.Events.LogEvent})" />,
/// not both.
/// </remarks>
protected override async Task EmitBatchAsync(IEnumerable<LogEvent> events)
public async Task EmitBatchAsync(IEnumerable<LogEvent> events)
{
DynamicResponse result;

Expand All @@ -71,42 +84,32 @@ protected override async Task EmitBatchAsync(IEnumerable<LogEvent> events)
HandleResponse(events, result);
}

/// <summary>
/// Emit a batch of log events, running to completion synchronously.
/// </summary>
/// <param name="events">The events to emit.</param>
/// <returns>Response from Elasticsearch</returns>
protected virtual Task<T> EmitBatchCheckedAsync<T>(IEnumerable<LogEvent> events) where T : class, IElasticsearchResponse, new()
public Task OnEmptyBatchAsync()
{
// ReSharper disable PossibleMultipleEnumeration
if (events == null || !events.Any())
return Task.FromResult<T>(default(T));

var payload = CreatePayload(events);
return _state.Client.BulkAsync<T>(PostData.MultiJson(payload));
return Task.CompletedTask;
}

/// <summary>
/// Emit a batch of log events, running to completion synchronously.
/// </summary>
/// <param name="events">The events to emit.</param>
/// <returns>Response from Elasticsearch</returns>
protected virtual T EmitBatchChecked<T>(IEnumerable<LogEvent> events) where T : class, IElasticsearchResponse, new()
private Task<T> EmitBatchCheckedAsync<T>(IEnumerable<LogEvent> events) where T : class, IElasticsearchResponse, new()
{
// ReSharper disable PossibleMultipleEnumeration
if (events == null || !events.Any())
return null;
return Task.FromResult<T>(default(T));

var payload = CreatePayload(events);
return _state.Client.Bulk<T>(PostData.MultiJson(payload));
return _state.Client.BulkAsync<T>(PostData.MultiJson(payload));
}

/// <summary>
/// Handles the exceptions.
/// </summary>
/// <param name="ex"></param>
/// <param name="events"></param>
protected virtual void HandleException(Exception ex, IEnumerable<LogEvent> events)
private void HandleException(Exception ex, IEnumerable<LogEvent> events)
{
if (_state.Options.EmitEventFailure.HasFlag(EmitEventFailureHandling.WriteToSelfLog))
{
Expand Down Expand Up @@ -153,18 +156,6 @@ protected virtual void HandleException(Exception ex, IEnumerable<LogEvent> event
throw ex;
}

// Helper function: checks if a given dynamic member / dictionary key exists at runtime
private static bool HasProperty(dynamic settings, string name)
{
if (settings is IDictionary<string, object>)
return ((IDictionary<string, object>)settings).ContainsKey(name);

if (settings is System.Dynamic.DynamicObject)
return ((System.Dynamic.DynamicObject)settings).GetDynamicMemberNames().Contains(name);

return settings.GetType().GetProperty(name) != null;
}

private IEnumerable<string> CreatePayload(IEnumerable<LogEvent> events)
{
if (!_state.TemplateRegistrationSuccess && _state.Options.RegisterTemplateFailure == RegisterTemplateRecovery.FailSink)
Expand Down Expand Up @@ -280,7 +271,7 @@ internal static object CreateElasticAction(ElasticOpType opType, string indexNam
: new ElasticIndexAction(actionPayload);
return action;
}

sealed class ElasticCreateAction
{
public ElasticCreateAction(ElasticActionPayload payload)
Expand Down
Loading

0 comments on commit fa5b097

Please sign in to comment.