Disable cleanup_timeout by default in docker and kubernetes autodisco…

…ver (#24681) (#24730)

It is kept to 60 seconds in Filebeat, to give a time to collect logs.

Keeping configurations running for some time after containers have stopped
is needed in some cases to complete the collection of logs. But in the rest of
cases it is not usually needed, and leads to errors when querying endpoints
known to be down.
It can also lead to query IPs that are being reused in newer containers, what
can be misleading if the newer pod answers because these events will still
have the metadata of the old container.

(cherry picked from commit 439b808)

CHANGELOG.next.asciidoc

@@ -20,6 +20,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Make error message about locked data path actionable. {pull}18667[18667]
 - Fix panic with inline SSL when the certificate or key were small than 256 bytes. {pull}23820[23820]
 - Use alias to report container image in k8s metadata. {pull}24380[24380]
+- Set `cleanup_timeout` to zero by default in docker and kubernetes autodiscover in all beats except Filebeat where it is kept to 60 seconds. {pull}24681[24681]
 
 *Auditbeat*
 

filebeat/autodiscover/defaults.go

@@ -0,0 +1,30 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package autodiscover
+
+import (
+	"time"
+
+	"github.com/elastic/beats/v7/libbeat/autodiscover/providers/docker"
+	"github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes"
+)
+
+func init() {
+	docker.DefaultCleanupTimeout = 60 * time.Second
+	kubernetes.DefaultCleanupTimeout = 60 * time.Second
+}

filebeat/autodiscover/imports.go

@@ -0,0 +1,22 @@
+// Licensed to Elasticsearch B.V. under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Elasticsearch B.V. licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package autodiscover
+
+import (
+	_ "github.com/elastic/beats/v7/filebeat/autodiscover/builder/hints"
+)

filebeat/beater/filebeat.go

@@ -56,8 +56,8 @@ import (
 	_ "github.com/elastic/beats/v7/filebeat/processor/add_kubernetes_metadata"
 	_ "github.com/elastic/beats/v7/libbeat/processors/decode_csv_fields"
 
-	// include all filebeat specific builders
-	_ "github.com/elastic/beats/v7/filebeat/autodiscover/builder/hints"
+	// include all filebeat specific autodiscover features
+	_ "github.com/elastic/beats/v7/filebeat/autodiscover"
 )
 
 const pipelinesWarning = "Filebeat is unable to load the Ingest Node pipelines for the configured" +

libbeat/autodiscover/providers/docker/config.go

@@ -40,12 +40,15 @@ type Config struct {
 	CleanupTimeout time.Duration           `config:"cleanup_timeout" validate:"positive"`
 }
 
+// Public variable, so specific beats (as Filebeat) can set a different cleanup timeout if they need it.
+var DefaultCleanupTimeout time.Duration = 0
+
 func defaultConfig() *Config {
 	return &Config{
 		Host:           "unix:///var/run/docker.sock",
 		Prefix:         "co.elastic",
 		Dedot:          true,
-		CleanupTimeout: 60 * time.Second,
+		CleanupTimeout: DefaultCleanupTimeout,
 	}
 }
 

libbeat/autodiscover/providers/kubernetes/config.go

@@ -57,11 +57,14 @@ type Config struct {
 	AddResourceMetadata *metadata.AddResourceMetadataConfig `config:"add_resource_metadata"`
 }
 
+// Public variable, so specific beats (as Filebeat) can set a different cleanup timeout if they need it.
+var DefaultCleanupTimeout time.Duration = 0
+
 func defaultConfig() *Config {
 	return &Config{
 		SyncPeriod:     10 * time.Minute,
 		Resource:       "pod",
-		CleanupTimeout: 60 * time.Second,
+		CleanupTimeout: DefaultCleanupTimeout,
 		Prefix:         "co.elastic",
 		Unique:         false,
 	}

libbeat/docs/shared-autodiscover.asciidoc

@@ -117,7 +117,13 @@ It has the following settings:
 `ssl`:: (Optional) SSL configuration to use when connecting to the Docker
 socket.
 `cleanup_timeout`:: (Optional) Specify the time of inactivity before stopping the
-running configuration for a container, 60s by default.
+running configuration for a container,
+ifeval::["{beatname_lc}"=="filebeat"]
+ 60s by default.
+endif::[]
+ifeval::["{beatname_lc}"!="filebeat"]
+ disabled by default.
+endif::[]
 `labels.dedot`:: (Optional) Default to be false. If set to true, replace dots in
  labels with `_`.
 
@@ -217,7 +223,13 @@ The `kubernetes` autodiscover provider has the following configuration settings:
   namespaces. It is unset by default. The namespace configuration only applies to
   kubernetes resources that are namespace scoped.
 `cleanup_timeout`:: (Optional) Specify the time of inactivity before stopping the
-running configuration for a container, 60s by default.
+running configuration for a container,
+ifeval::["{beatname_lc}"=="filebeat"]
+ 60s by default.
+endif::[]
+ifeval::["{beatname_lc}"!="filebeat"]
+ disabled by default.
+endif::[]
 `kube_config`:: (Optional) Use given config file as configuration for Kubernetes
   client. If kube_config is not set, KUBECONFIG environment variable will be
   checked and if not present it will fall back to InCluster.