Using durable seq (#205)

* added BufferFileCountLimit to limit number of files option to set number of days of logfiles to keep #202 * init * Added example and fixes for persistant logging mosly code from https://github.com/serilog/serilog-sinks-seq * added BufferCleanPayload * dont force inheritance the timer start in base class is ca 5sec and will not start before inheritade base class is finished * renamed to Elasticsearch and added documentation in Readme and chenges * rename
serilog-contrib · Jan 26, 2019 · 4f3a7ab · 4f3a7ab
1 parent e03cb1e
commit 4f3a7ab
Show file tree

Hide file tree

Showing 31 changed files with 1,952 additions and 648 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,5 +1,11 @@
 == Changelog
-
+7.0
+ * DurableElasticsearchSink is rewritten to use the same base code as the sink for Serilog.Sinks.Seq. Nuget Serilog.Sinks.File is now used instead of deprecated Serilog.Sinks.RollingFile. Lots of new fintuning options for file storage is added in ElasticsearchSinkOptions.  Updated  Serilog.Sinks.Elasticsearch.Sample.Main with SetupLoggerWithPersistantStorage with all available options for durable mode.
+ * Changed datatype on singleEventSizePostingLimit  from int to long? with default value null. to make it possible ro reuse code from Sinks.Seq .
+ * IndexDecider didnt worked well in buffer mode because of LogEvent was null. Added BufferIndexDecider.
+ * Added BufferCleanPayload and an example which makes it possible to cleanup your invalid logging document if rejected from elastic because of inconsistent datatype on a field. It'seasy to miss errors in the self log now its possible to se logrows which is bad for elasticsearch in the elastic log.
+ * Added BufferRetainedInvalidPayloadsLimitBytes A soft limit for the number of bytes to use for storing failed requests.
+ * Added BufferFileCountLimit The maximum number of log files that will be retained.
 6.4
  * Render message by default (#160). 
  * Expose interface-typed options via appsettings (#162)

diff --git a/README.md b/README.md
@@ -49,6 +49,8 @@ This example shows the options that are currently available when using the appSe
     <add key="serilog:write-to:Elasticsearch.bufferBaseFilename" value="C:\Temp\SerilogElasticBuffer"/>
     <add key="serilog:write-to:Elasticsearch.bufferFileSizeLimitBytes" value="5242880"/>
     <add key="serilog:write-to:Elasticsearch.bufferLogShippingInterval" value="5000"/>	
+    <add key="serilog:write-to:Elasticsearch.bufferRetainedInvalidPayloadsLimitBytes" value="5000"/>	
+    <add key="serilog:write-to:Elasticsearch.bufferFileCountLimit " value="31"/>	     
     <add key="serilog:write-to:Elasticsearch.connectionGlobalHeaders" value="Authorization=Bearer SOME-TOKEN;OtherHeader=OTHER-HEADER-VALUE" />
     <add key="serilog:write-to:Elasticsearch.connectionTimeout" value="5" />
     <add key="serilog:write-to:Elasticsearch.emitEventFailure" value="WriteToSelfLog" />
@@ -147,6 +149,8 @@ In your `appsettings.json` file, under the `Serilog` node, :
           "bufferBaseFilename":  "C:/Temp/LogDigipolis/docker-elk-serilog-web-buffer",
           "bufferFileSizeLimitBytes": 5242880,
           "bufferLogShippingInterval": 5000,
+          "bufferRetainedInvalidPayloadsLimitBytes": 5000,
+          "bufferFileCountLimit": 31,
           "connectionGlobalHeaders" :"Authorization=Bearer SOME-TOKEN;OtherHeader=OTHER-HEADER-VALUE",
           "connectionTimeout": 5,
           "emitEventFailure": "WriteToSelfLog",
@@ -203,7 +207,37 @@ Since version 5.5 you can use the RegisterTemplateFailure option. Set it to one
  - IndexToDeadletterIndex; using the deadletterindex format, it will write the events to the deadletter queue. When you fix your template mapping, you can copy your data into the right index.
  - FailSink; this will simply fail the sink by raising an exception.
 
+ Since version 7 you can  specify an action to do when log row was denied by the elasticsearch because of the data (payload) if durable file is specied.
+ i.e.
+```csharp
+BufferCleanPayload = (failingEvent, statuscode, exception) =>
+                    {
+                        dynamic e = JObject.Parse(failingEvent);
+                        return JsonConvert.SerializeObject(new Dictionary<string, object>()
+                        {
+                            { "@timestamp",e["@timestamp"]},
+                            { "level","Error"},
+                            { "message","Error: "+e.message},
+                            { "messageTemplate",e.messageTemplate},
+                            { "failingStatusCode", statuscode},
+                            { "failingException", exception}
+                        });
+                    },
+
+```
+The IndexDecider didnt worked well when durable file was specified so an option to specify BufferIndexDecider is added.
+Datatype of logEvent is string
+i.e.
+```csharp
+ BufferIndexDecider = (logEvent, offset) => "log-serilog-" + (new Random().Next(0, 2)),
+```
+Option BufferFileCountLimit is added. The maximum number of log files that will be retained. including the current log file. For unlimited retention, pass null. The default is 31.
+Option BufferFileSizeLimitBytes is added The maximum size, in bytes, to which the buffer log file for a specific date will be allowed to grow. By default 100L * 1024 * 1024 will be applied.
+### Breaking changes for version 7
+Nuget Serilog.Sinks.File is now used instead of deprecated Serilog.Sinks.RollingFile
+SingleEventSizePostingLimit option is changed from int to long? with default value null, Don't use value 0 nothing will be logged then!!!!!
 
+
 ### Breaking changes for version 6
 
 Starting from version 6, the sink has been upgraded to work with Elasticsearch 6.0 and has support for the new templates used by ES 6. 

diff --git a/sample/Serilog.Sinks.Elasticsearch.Sample/Program.cs b/sample/Serilog.Sinks.Elasticsearch.Sample/Program.cs
@@ -1,36 +1,57 @@
 using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Reflection.Metadata.Ecma335;
+using System.Threading;
+using Microsoft.Extensions.Configuration;
+using Newtonsoft.Json;
+using Newtonsoft.Json.Linq;
 using Serilog;
+using Serilog.Core;
 using Serilog.Debugging;
+using Serilog.Events;
 using Serilog.Formatting.Json;
-using Serilog.Sinks.RollingFile;
+using Serilog.Sinks.File;
 using Serilog.Sinks.SystemConsole.Themes;
 
 namespace Serilog.Sinks.Elasticsearch.Sample
 {
     class Program
     {
+        private static IConfiguration Configuration { get; } = new ConfigurationBuilder()
+            .SetBasePath(Directory.GetCurrentDirectory())
+            .AddJsonFile("appsettings.json", true, true)
+            .AddEnvironmentVariables()
+            .Build();
         static void Main(string[] args)
         {
+
+            // Enable the selflog output
+            SelfLog.Enable(Console.Error);
             Log.Logger = new LoggerConfiguration()
                 .MinimumLevel.Debug()
                 .WriteTo.Console(theme: SystemConsoleTheme.Literate)
-                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://elastic:changeme@localhost:9200")) // for the docker-compose implementation
-                {
-                    AutoRegisterTemplate = true,
-                    //BufferBaseFilename = "./buffer",
-                    RegisterTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
-                    FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
-                    EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
-                                       EmitEventFailureHandling.WriteToFailureSink |
-                                       EmitEventFailureHandling.RaiseCallback,
-                    FailureSink = new RollingFileSink("./fail-{Date}.txt", new JsonFormatter(), null, null)
-                })
-                .CreateLogger();
+                                //not persistant
+                                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(Configuration.GetConnectionString("elasticsearch"))) // for the docker-compose implementation
+                                {
+                                    AutoRegisterTemplate = true,
+                                    //BufferBaseFilename = "./buffer",
+                                    RegisterTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
+                                    FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
+                                    EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
+                                                       EmitEventFailureHandling.WriteToFailureSink |
+                                                       EmitEventFailureHandling.RaiseCallback,
+                                    FailureSink = new FileSink("./fail-{Date}.txt", new JsonFormatter(), null, null)
+                                })
+                    .CreateLogger();
 
-            // Enable the selflog output
-            SelfLog.Enable(Console.Error);
+            //SetupLoggerWithSimplePersistantStorage();
+            //LoggingLevelSwitch levelSwitch = SetupLoggerWithPersistantStorage();
 
+            //Log.Debug("To high loglevel default is Information this will not be logged");
+            //levelSwitch.MinimumLevel = LogEventLevel.Debug;
             Log.Information("Hello, world!");
+            //Log.Information("To  big log row bigger than SingleEventSizePostingLimit ! {a}", new string('*', 5000));
 
             int a = 10, b = 0;
             try
@@ -47,7 +68,70 @@ static void Main(string[] args)
             Log.Debug("Reusing {A} by {B}", "string", true);
 
             Log.CloseAndFlush();
-            Console.Read();
+            Console.WriteLine("Press any key to continue...");
+            while (!Console.KeyAvailable)
+            {
+                Thread.Sleep(500);
+            }
+        }
+
+        private static LoggingLevelSwitch SetupLoggerWithPersistantStorage()
+        {
+            //persistant storage with all settings available for this mode
+            //please note that all limit settings here is set verry low for test, default values should usually work best!
+            var levelSwitch = new LoggingLevelSwitch();
+            Log.Logger = new LoggerConfiguration()
+                .MinimumLevel.Debug()
+                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(Configuration.GetConnectionString("elasticsearch")))
+                {
+                    AutoRegisterTemplate = true,
+                    BufferBaseFilename = "./buffer/logserilog",
+                    IndexFormat = "log-serilog-{0:yyyy.MM}",
+                    RegisterTemplateFailure = RegisterTemplateRecovery.FailSink,
+                    BufferCleanPayload = (failingEvent, statuscode, exception) =>
+                    {
+                        dynamic e = JObject.Parse(failingEvent);
+                        return JsonConvert.SerializeObject(new Dictionary<string, object>()
+                        {
+                            { "@timestamp",e["@timestamp"]},
+                            { "level",e.level},
+                            { "message","Error: "+e.message},
+                            { "messageTemplate",e.messageTemplate},
+                            { "failingStatusCode", statuscode},
+                            { "failingException", exception}
+                        });
+                    },
+                    OverwriteTemplate = true,
+                    NumberOfShards = 1,
+                    NumberOfReplicas = 1,
+                    GetTemplateContent = null,
+                    AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv6,
+                    PipelineName = null,
+                    TypeName = "logevent",
+                    BufferIndexDecider = (logEvent, offset) => "log-serilog-" + (new Random().Next(0, 2)),
+                    BatchPostingLimit = 50,
+                    BufferLogShippingInterval = TimeSpan.FromSeconds(5),
+                    SingleEventSizePostingLimit = 1000,
+                    LevelSwitch = levelSwitch,
+                    BufferRetainedInvalidPayloadsLimitBytes = 2000,
+                    BufferFileSizeLimitBytes = 2000,
+                    BufferFileCountLimit = 2
+                })
+                .CreateLogger();
+            return levelSwitch;
+        }
+
+        private static void SetupLoggerWithSimplePersistantStorage()
+        {
+            //presistant
+            Log.Logger = new LoggerConfiguration()
+                .MinimumLevel.Debug()
+                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(Configuration.GetConnectionString("elasticsearch")))
+                {
+                    BufferBaseFilename = "./buffer/logserilogsimple",
+                    IndexFormat = "log-serilog-simple-{0:yyyy.MM}"
+                })
+                .CreateLogger();
         }
     }
 }
diff --git a/sample/Serilog.Sinks.Elasticsearch.Sample/Serilog.Sinks.Elasticsearch.Sample.csproj b/sample/Serilog.Sinks.Elasticsearch.Sample/Serilog.Sinks.Elasticsearch.Sample.csproj
@@ -1,11 +1,15 @@
-<Project Sdk="Microsoft.NET.Sdk">
+<Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
     <OutputType>Exe</OutputType>
-    <TargetFramework>netcoreapp1.1</TargetFramework>
+    <TargetFramework>netcoreapp2.1</TargetFramework>
   </PropertyGroup>
 
   <ItemGroup>
+    <PackageReference Include="Microsoft.Extensions.Configuration" Version="2.1.1" />
+    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="2.1.1" />
+    <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="2.1.1" />
+    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="2.1.1" />
     <PackageReference Include="Serilog" Version="2.6.0" />
     <PackageReference Include="Serilog.Sinks.Console" Version="3.1.1" />
   </ItemGroup>
@@ -14,4 +18,10 @@
     <ProjectReference Include="..\..\src\Serilog.Sinks.Elasticsearch\Serilog.Sinks.Elasticsearch.csproj" />
   </ItemGroup>
 
+  <ItemGroup>
+    <None Update="appsettings.json">
+      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
+    </None>
+  </ItemGroup>
+
 </Project>
diff --git a/sample/Serilog.Sinks.Elasticsearch.Sample/appsettings.json b/sample/Serilog.Sinks.Elasticsearch.Sample/appsettings.json
@@ -0,0 +1,5 @@
+{
+    "ConnectionStrings": {
+        "elasticsearch": "http://elastic:changeme@localhost:9200"
+    }
+}
diff --git a/src/Serilog.Sinks.Elasticsearch/LoggerConfigurationElasticSearchExtensions.cs b/src/Serilog.Sinks.Elasticsearch/LoggerConfigurationElasticSearchExtensions.cs
@@ -24,6 +24,7 @@
 using System.ComponentModel;
 using Elasticsearch.Net;
 using Serilog.Formatting;
+using Serilog.Sinks.Elasticsearch.Durable;
 
 namespace Serilog
 {
@@ -119,6 +120,7 @@ public static LoggerConfiguration Elasticsearch(
         /// <param name="levelSwitch">A switch allowing the pass-through minimum level to be changed at runtime.</param>
         /// <param name="bufferBaseFilename"><see cref="ElasticsearchSinkOptions.BufferBaseFilename"/></param>
         /// <param name="bufferFileSizeLimitBytes"><see cref="ElasticsearchSinkOptions.BufferFileSizeLimitBytes"/></param>
+        /// <param name="bufferFileCountLimit"><see cref="ElasticsearchSinkOptions.BufferFileCountLimit"/></param>        
         /// <param name="bufferLogShippingInterval"><see cref="ElasticsearchSinkOptions.BufferLogShippingInterval"/></param>
         /// <param name="connectionGlobalHeaders">A comma or semi column separated list of key value pairs of headers to be added to each elastic http request</param>   
         /// <param name="connectionTimeout"><see cref="ElasticsearchSinkOptions.ConnectionTimeout"/>The connection timeout (in seconds) when sending bulk operations to elasticsearch (defaults to 5).</param>   
@@ -139,7 +141,7 @@ public static LoggerConfiguration Elasticsearch(
         /// <param name="customFormatter">Customizes the formatter used when converting log events into ElasticSearch documents. Please note that the formatter output must be valid JSON :)</param>
         /// <param name="customDurableFormatter">Customizes the formatter used when converting log events into the durable sink. Please note that the formatter output must be valid JSON :)</param>
         /// <param name="failureSink">Sink to use when Elasticsearch is unable to accept the events. This is optionally and depends on the EmitEventFailure setting.</param>   
-        /// <param name="singleEventSizePostingLimit"><see cref="ElasticsearchSinkOptions.SingleEventSizePostingLimit"/>The maximum length of an event allowed to be posted to Elasticsearch.</param>        
+        /// <param name="singleEventSizePostingLimit"><see cref="ElasticsearchSinkOptions.SingleEventSizePostingLimit"/>The maximum length of an event allowed to be posted to Elasticsearch.default null</param>        
         /// <returns>LoggerConfiguration object</returns>
         /// <exception cref="ArgumentNullException"><paramref name="nodeUris"/> is <see langword="null" />.</exception>
         public static LoggerConfiguration Elasticsearch(
@@ -153,7 +155,7 @@ public static LoggerConfiguration Elasticsearch(
             bool inlineFields = false,
             LogEventLevel restrictedToMinimumLevel = LevelAlias.Minimum,
             string bufferBaseFilename = null,
-            long? bufferFileSizeLimitBytes = null,
+            long? bufferFileSizeLimitBytes = null,            
             long bufferLogShippingInterval = 5000,
             string connectionGlobalHeaders = null,
             LoggingLevelSwitch levelSwitch = null,
@@ -175,7 +177,8 @@ public static LoggerConfiguration Elasticsearch(
             ITextFormatter customFormatter = null,
             ITextFormatter customDurableFormatter = null,
             ILogEventSink failureSink = null,
-            int singleEventSizePostingLimit = 0)
+            long? singleEventSizePostingLimit = null,
+            int? bufferFileCountLimit = null)
         {
             if (string.IsNullOrEmpty(nodeUris))
                 throw new ArgumentNullException(nameof(nodeUris), "No Elasticsearch node(s) specified.");
@@ -220,6 +223,10 @@ public static LoggerConfiguration Elasticsearch(
                 options.BufferFileSizeLimitBytes = bufferFileSizeLimitBytes.Value;
             }
 
+            if (bufferFileCountLimit.HasValue)
+            {
+                options.BufferFileCountLimit = bufferFileCountLimit.Value;
+            }
             options.BufferLogShippingInterval = TimeSpan.FromMilliseconds(bufferLogShippingInterval);
 
             if (!string.IsNullOrWhiteSpace(connectionGlobalHeaders))

diff --git a/src/Serilog.Sinks.Elasticsearch/Serilog.Sinks.Elasticsearch.csproj b/src/Serilog.Sinks.Elasticsearch/Serilog.Sinks.Elasticsearch.csproj
@@ -28,23 +28,43 @@
     <GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
   </PropertyGroup>
 
+  <ItemGroup>
+    <Compile Remove="Sinks\ElasticSearch\Durable\ElasticSearch\ElasticSearchLogShipperOld.cs" />
+  </ItemGroup>
+
   <ItemGroup>
     <PackageReference Include="Elasticsearch.Net" Version="6.0.0" />
     <PackageReference Include="Serilog" Version="2.6.0" />
+    <PackageReference Include="Serilog.Formatting.Compact" Version="1.0.0" />
     <PackageReference Include="Serilog.Sinks.File" Version="4.0.0" />
     <PackageReference Include="Serilog.Sinks.PeriodicBatching" Version="2.1.1" />
-    <PackageReference Include="Serilog.Sinks.RollingFile" Version="3.3.0" />
   </ItemGroup>
 
   <ItemGroup Condition=" '$(TargetFramework)' == 'net45' ">
     <Reference Include="System" />
     <Reference Include="Microsoft.CSharp" />
   </ItemGroup>
 
+    <PropertyGroup Condition=" '$(TargetFramework)' == 'netstandard1.3' ">
+        <NoWarn>1591;1701;1702</NoWarn>
+        <DefineConstants>$(DefineConstants);DURABLE;THREADING_TIMER</DefineConstants>
+    </PropertyGroup>
+
+    <PropertyGroup Condition=" '$(TargetFramework)' == 'net45' ">
+        <NoWarn>1591;1701;1702</NoWarn>
+        <DefineConstants>$(DefineConstants);DURABLE;THREADING_TIMER;HRESULTS</DefineConstants>
+    </PropertyGroup>
+
   <PropertyGroup Condition=" '$(TargetFramework)' == 'netstandard1.3' ">
+      <NoWarn>1591;1701;1702</NoWarn>
     <DefineConstants>$(DefineConstants);DOTNETCORE;NO_SERIALIZATION;NO_TIMER</DefineConstants>
   </PropertyGroup>
 
+  <PropertyGroup Condition="'$(Configuration)|$(TargetFramework)|$(Platform)'=='Debug|net45|AnyCPU'">
+    <NoWarn>1591;1701;1702</NoWarn>
+    <WarningsAsErrors>NU1605</WarningsAsErrors>
+  </PropertyGroup>
+
   <ItemGroup>
     <DotNetCliToolReference Include="dotnet-version" Version="1.1.0" />
   </ItemGroup>