Deprecate config_dir (Filebeat) and refactor docs (elastic#5321) (ela…

…stic#5331)

This deprecates the config_dir option in Filebeat, as it's functionality is
covered by `filebeat.config.prospectors`. There's a bigger refactoring in the
docs, to reflect that the `filebeat.config.*` and `metricbeat.config.*` options
can be used for both including external files and live reloading. Only the
live reloading part stays in Beta, including external config files is GA.

Part of elastic#4581.

(cherry picked from commit 9af1d5c)

CHANGELOG.asciidoc

@@ -70,6 +70,8 @@ https://github.com/elastic/beats/compare/v6.0.0-rc1...master[Check the HEAD diff
 
 *Filebeat*
 
+- The `filebeat.config_dir` option is deprecated. Use `filebeat.config.prospector` options instead. {pull}5321[5321]
+
 *Heartbeat*
 
 *Metricbeat*

filebeat/config/config.go

@@ -9,6 +9,7 @@ import (
 
 	"github.com/elastic/beats/libbeat/cfgfile"
 	"github.com/elastic/beats/libbeat/common"
+	"github.com/elastic/beats/libbeat/common/cfgwarn"
 	"github.com/elastic/beats/libbeat/logp"
 	"github.com/elastic/beats/libbeat/paths"
 )
@@ -94,6 +95,8 @@ func (config *Config) FetchConfigs() error {
 		return nil
 	}
 
+	cfgwarn.Deprecate("7.0.0", "config_dir is deprecated. Use `filebeat.config.prospectors` instead.")
+
 	// If configDir is relative, consider it relative to the config path
 	configDir = paths.Resolve(paths.Config, configDir)
 

filebeat/docs/reload-configuration.asciidoc

@@ -1,39 +1,16 @@
 [[filebeat-configuration-reloading]]
-== Reload the configuration dynamically
-
-beta[]
-
-You can configure Filebeat to dynamically reload configuration files when there
-are changes. This feature is available for prospector and module configuration
-files only.
-
-To configure this feature, you specify a path
-(https://golang.org/pkg/path/filepath/#Glob[Glob]) to watch for configuration
-changes. When the files found by the Glob change, new prospectors and/or
-modules are started and stopped according to changes in the configuration files.
-
-This feature is especially useful in container environments where one container
-is used to tail logs for services running in other containers on the same host.
-
-To enable dynamic config reloading, you specify the following options under
-`filebeat.config.prospectors` (for prospector configs) or
-`filebeat.config.modules` (for module configs):
-
-`path`:: A Glob that defines the files to check for changes.
-`reload.enabled`:: When set to `true`, enables dynamic config reload.
-`reload.period`:: Specifies how often the files are checked for changes. Do not
-set the `period` to less than 1s because the modification time of files is often
-stored in seconds. Setting the `period` to less than 1s will result in
-unnecessary overhead.
-
-See <<reload-prospector-config>> and <<reload-module-config>> for examples.
+== Load external configuration files
 
+Filebeat can load external configuration files for prospectors and modules,
+which allows you to separate your configuration into multiple smaller
+configuration files. See the <<load-prospector-config>> and the
+<<load-module-config>> sections for details.
 
 [float]
-[[reload-prospector-config]]
+[[load-prospector-config]]
 === Prospector config
 
-For prospector configurations, you specify reload options in the
+For prospector configurations, you specify `path` option in the
 `filebeat.config.prospectors` section of the +{beatname_lc}.yml+ file. For
 example:
 
@@ -42,8 +19,6 @@ example:
 filebeat.config.prospectors:
   enabled: true
   path: configs/*.yml
-  reload.enabled: true
-  reload.period: 10s
 ------------------------------------------------------------------------------
 
 Each file found by the Glob must contain a list of one or more prospector
@@ -68,20 +43,19 @@ file paths defined. If more than one prospector harvests the same file at the
 same time, it can lead to unexpected behaviour.
 
 [float]
-[[reload-module-config]]
+[[load-module-config]]
 === Module config
 
-For module configurations, you specify reload options in the
+For module configurations, you specify the `path` option in the
 `filebeat.config.modules` section of the +{beatname_lc}.yml+ file. For example:
 
 [source,yaml]
 ------------------------------------------------------------------------------
 filebeat.config.modules:
   enabled: true
   path: prospectors.d/*.yml <1>
-  reload.enabled: true
-  reload.period: 10s
 ------------------------------------------------------------------------------
+
 <1> If you change the path setting to look for config changes in a different
 directory, you will not be able to use the <<modules-command,`modules`>> command
 to enable and disable module configurations.
@@ -99,3 +73,39 @@ definitions. For example:
     enabled: true
     var.paths: [/var/log/apache2/error.log*]
 ------------------------------------------------------------------------------
+
+=== Live reloading
+
+beta[]
+
+You can configure Filebeat to dynamically reload configuration files when there
+are changes. This feature is available for prospector and module configuration
+files only.
+
+To configure this feature, you specify a path
+(https://golang.org/pkg/path/filepath/#Glob[Glob]) to watch for configuration
+changes. When the files found by the Glob change, new prospectors and/or
+modules are started and stopped according to changes in the configuration files.
+
+This feature is especially useful in container environments where one container
+is used to tail logs for services running in other containers on the same host.
+
+To enable dynamic config reloading, you specify the `path` and `reload` options
+under `filebeat.config.prospectors` or `filebeat.config.modules` sections. For
+example:
+
+[source,yaml]
+------------------------------------------------------------------------------
+filebeat.config.prospectors:
+  enabled: true
+  path: configs/*.yml
+  reload.enabled: true
+  reload.period: 10s
+------------------------------------------------------------------------------
+
+`path`:: A Glob that defines the files to check for changes.
+`reload.enabled`:: When set to `true`, enables dynamic config reload.
+`reload.period`:: Specifies how often the files are checked for changes. Do not
+set the `period` to less than 1s because the modification time of files is often
+stored in seconds. Setting the `period` to less than 1s will result in
+unnecessary overhead.

metricbeat/docs/reload-configuration.asciidoc

@@ -1,5 +1,36 @@
 [[metricbeat-configuration-reloading]]
-== Reload the configuration dynamically
+== Load external configuration files
+
+Metricbeat can load external configuration files for modules, which allows you
+to separate your configuration into multiple smaller configuration files. To use
+this, you specify the `path` option under `metricbeat.config.modules` in the
+main `metricbeat.yml` configuration file. For example:
+
+[source,yaml]
+------------------------------------------------------------------------------
+metricbeat.config.modules:
+  path: configs/*.yml
+------------------------------------------------------------------------------
+
+`path`:: A Glob that defines the files to check for changes.
+
+Each file found by the Glob must contain a list of one or more module
+definitions. For example:
+
+[source,yaml]
+------------------------------------------------------------------------------
+- module: system
+  metricsets: ["cpu"]
+  enabled: false
+  perdiod: 1s
+
+- module: system
+  metricsets: ["network"]
+  enabled: true
+  period: 10s
+------------------------------------------------------------------------------
+
+=== Live reloading
 
 beta[]
 
@@ -34,18 +65,3 @@ set the `period` to less than 1s because the modification time of files is often
 stored in seconds. Setting the `period` to less than 1s will result in
 unnecessary overhead.
 
-Each file found by the Glob must contain a list of one or more module
-definitions. For example:
-
-[source,yaml]
-------------------------------------------------------------------------------
-- module: system
-  metricsets: ["cpu"]
-  enabled: false
-  perdiod: 1s
-
-- module: system
-  metricsets: ["network"]
-  enabled: true
-  period: 10s
-------------------------------------------------------------------------------