[8.16](backport #41640) libbeat: increase total_fields.limit to 12500 (…

…#41656) It increased the `index.mapping.total_fields.limit` from `10000` to `12500` in order to avoid ingestion failures caused by too many field in the index. Since 8.15.0 the limit started to be hit. The field count being exceeded is on the index, counting all mapped fields and the dynamic fields. That's why a small event might trigger the error, the event contains new fields to be mapped which would exceed the total field limit if mapped. (cherry picked from commit 42dd93b) --------- Co-authored-by: Anderson Queiroz <[email protected]>
elastic · Nov 22, 2024 · 9d96bf4 · 9d96bf4
1 parent 3242411
commit 9d96bf4
Show file tree

Hide file tree

Showing 3 changed files with 4 additions and 3 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -10,7 +10,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 
 *Affecting all Beats*
 
-- Fixes filestream logging the error "filestream input with ID 'ID' already exists, this will lead to data duplication[...]" on Kubernetes when using autodiscover. {pull}41585[41585]
+- Fix metrics not being ingested, due to "Limit of total fields [10000] has been exceeded while adding new fields [...]". The total fields limit has been increased to 12500. No significant performance impact on Elasticsearch is anticipated. {pull}41640[41640]
 
 *Auditbeat*
 
@@ -27,6 +27,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Removed deprecated Cylance from Beats. See <<migrate-from-deprecated-module>> for migration options. {pull}38037[38037]
 - Removed deprecated Bluecoat from Beats. See <<migrate-from-deprecated-module>> for migration options. {pull}38037[38037]
 - Introduce input/netmetrics and refactor netflow input metrics {pull}38055[38055]
+- Fixes filestream logging the error "filestream input with ID 'ID' already exists, this will lead to data duplication[...]" on Kubernetes when using autodiscover. {pull}41585[41585]
 
 
 *Heartbeat*

diff --git a/libbeat/template/load_test.go b/libbeat/template/load_test.go
@@ -170,7 +170,7 @@ func TestFileLoader_Load(t *testing.T) {
 							"refresh_interval": "5s",
 							"mapping": mapstr.M{
 								"total_fields": mapstr.M{
-									"limit": 10000,
+									"limit": defaultTotalFieldsLimit,
 								},
 							},
 							"query": mapstr.M{

diff --git a/libbeat/template/template.go b/libbeat/template/template.go
@@ -35,7 +35,7 @@ import (
 var (
 	// Defaults used in the template
 	defaultDateDetection           = false
-	defaultTotalFieldsLimit        = 10000
+	defaultTotalFieldsLimit        = 12500
 	defaultMaxDocvalueFieldsSearch = 200
 
 	defaultFields []string