perf: tweak schema sampling rate (#2252)

Logflare · Nov 5, 2024 · dcecba1 · dcecba1
1 parent f571b9b
commit dcecba1
Show file tree

Hide file tree

Showing 2 changed files with 5 additions and 5 deletions.
diff --git a/docs/docs.logflare.com/docs/concepts/ingestion/index.mdx b/docs/docs.logflare.com/docs/concepts/ingestion/index.mdx
@@ -104,7 +104,7 @@ metadata: {
 :::note
 On high ingestion volume, Logflare will sample incoming events instead of checking each event. The sample rate decreases as the ingestion rate increases. Ingestion rates are compared only on an individual local server that is performing the ingestion.
 
-From 10-100 events per second, sample rate is 0.2. From 100-500 events per second, sample rate is 0.1. From 500-1000 events per second, sample rate is 0.05. Above 1000 events per second, sample rate is 0.01.
+From 10-100 events per second, sample rate is 0.1. From 100-1,000 events per second, sample rate is 0.01. From 1,000-10,000 events per second, sample rate is 0.001. Above 10,000 events per second, sample rate is 0.0001.
 :::
 
 ### Key Transformation

diff --git a/lib/logflare/source/bigquery/pipeline.ex b/lib/logflare/source/bigquery/pipeline.ex
@@ -213,10 +213,10 @@ defmodule Logflare.Source.BigQuery.Pipeline do
     # random sample if local ingest rate is above a certain level
     probability =
       case PubSubRates.Cache.get_local_rates(source.token) do
-        %{average_rate: avg} when avg > 1000 -> 0.01
-        %{average_rate: avg} when avg > 500 -> 0.05
-        %{average_rate: avg} when avg > 100 -> 0.1
-        %{average_rate: avg} when avg > 10 -> 0.2
+        %{average_rate: avg} when avg > 10000 -> 0.0001
+        %{average_rate: avg} when avg > 1000 -> 0.001
+        %{average_rate: avg} when avg > 100 -> 0.01
+        %{average_rate: avg} when avg > 10 -> 0.1
         _ -> 1
       end