Skip to content

Commit

Permalink
adds edits for the migration topic and some misc fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
dedemorton authored and Tudor Golubenco committed Nov 23, 2015
1 parent 064d7eb commit 6557682
Show file tree
Hide file tree
Showing 6 changed files with 64 additions and 55 deletions.
20 changes: 10 additions & 10 deletions docs/command-line.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,21 @@ configuration file.
----------------------------------------------------------------------------
$ ./filebeat -h
Usage of ./filebeat:
-N Disable actual publishing for testing
-N Disable actual publishing for testing
-c string
Configuration file (default "/etc/filebeat/filebeat.yml")
Configuration file (default "/etc/filebeat/filebeat.yml")
-configtest
Test configuration and exit.
-cpuprofile string
Write cpu profile to file
Write cpu profile to file
-d string
Enable certain debug selectors
-e Output to stdout and disable syslog/file output
Enable certain debug selectors
-e Log to stderr and disable syslog/file output
-memprofile string
Write memory profile to this file
-test
Test configuration and exit.
-v Log at INFO level
Write memory profile to this file
-v Log at INFO level
-version
Print version and exit
Print version and exit
-----------------------------------------------------------------------------


Expand Down
18 changes: 11 additions & 7 deletions docs/configuration.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,25 @@
[[filebeat-configuration-details]]
== Configuration Options

Before modifying configuration settings, make sure you've read <<filebeat-configuration>>.

The Filebeat configuration file uses http://yaml.org/[YAML] for its syntax.
The _etc/filebeat.yml_ file consists of the following sections:
The file contains config options common to all Beats. These options are described
in the following sections of the {libbeat}/configuration.html[Beats Platform Reference]:

* {libbeat}/configuration.html#configuration-shipper[Shipper]
* {libbeat}/configuration.html#configuration-output[Output]
* <<configuration-filebeat-options>>
* {libbeat}/configuration.html#configuration-logging[Logging (optional)]
* {libbeat}/configuration.html#configuration-run-options[Run options (optional)]
* {libbeat}/configuration.html#configuration-logging[Logging (Optional)]
* {libbeat}/configuration.html#configuration-run-options[Run Options (Optional)]

The `shipper`, `output`, `logging`, and `runoptions` sections contain configuration options that are common to all Beats, so they are covered in the {libbeat}/configuration.html[Beats Platform Reference].
The Filebeat config options are described here:

* <<configuration-filebeat-options>>

[[configuration-filebeat-options]]
=== Prospectors
=== Filebeat

The `prospectors` section specifies a list of prospectors that Filebeat
The `filebeat` section specifies a list of `prospectors` that Filebeat
uses to locate and process log files. Each prospector item begins with a dash (-)
and specifies prospector-specific configuration options, including
the list of paths that are crawled to locate log files.
Expand Down
27 changes: 13 additions & 14 deletions docs/getting-started.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ After you have installed these products, you can start <<filebeat-installation>>
[[filebeat-installation]]
=== Installing Filebeat

To download and install Filebeat, run the following commands. Use the commands that
To download and install Filebeat, run the following commands. Use the commands that
work with your system (deb for Debian/Ubuntu, rpm for Redhat/Centos/Fedora, mac for OS X).

deb:
Expand Down Expand Up @@ -45,14 +45,14 @@ tar xzvf filebeat-{version}-darwin.tgz
NOTE: We also provide 32-bit images that you can get from our
https://www.elastic.co/downloads/beats/filebeat[download page].

Before starting filebeat, you should look at the configuration options in the configuration
file, for example `/etc/filebeat/filebeat.yml`. For more information about these options,
see <<filebeat-configuration-details,Configuration>>.
Before starting filebeat, you should look at the configuration options in the configuration
file, for example `/etc/filebeat/filebeat.yml`. For more information about these options,
see <<filebeat-configuration-details>>.

[[filebeat-configuration]]
=== Configuring Filebeat

To configure Filebeat, you edit the _filebeat.yml_ file. Here is a sample of
To configure Filebeat, you edit the _filebeat.yml_ file. Here is a sample of
the _filebeat.yml_ file:

[source,yaml]
Expand All @@ -77,8 +77,8 @@ filebeat:
type: log
-------------------------------------------------------------------------------------

Filebeat uses predefined default values for most configuration options. For the most basic
Filebeat configuration, you can define a single prospector with a single path. For example:
Filebeat uses predefined default values for most configuration options. For the most basic
Filebeat configuration, you can define a single prospector with a single path. For example:

[source,yaml]
-------------------------------------------------------------------------------------
Expand All @@ -89,7 +89,7 @@ filebeat:
- "/var/log/*.log"
-------------------------------------------------------------------------------------

The prospector in this example harvests all files in the path `/var/log/*.log`, which means
The prospector in this example harvests all files in the path `/var/log/*.log`, which means
that Filebeat will harvest all files in the directory `/var/log/` that end with `.log`. All patterns supported
by https://golang.org/pkg/path/filepath/#Glob[Golang Glob] are also supported here.

Expand Down Expand Up @@ -118,11 +118,10 @@ filebeat:
-------------------------------------------------------------------------------------

The config file in the example starts two prospectors. The first prospector has two harvesters,
one harvesting the `system.log` file, and the other harvesting `wifi.log`. The second prospector
one harvesting the `system.log` file, and the other harvesting `wifi.log`. The second prospector
starts a harvester for each file in the apache directory.


See <<filebeat-configuration-details,Configuration>> for more details about each configuration option.
See <<filebeat-configuration-details>> for more details about each configuration option.

[[filebeat-template]]
=== Loading the Index Template in Elasticsearch
Expand Down Expand Up @@ -154,8 +153,8 @@ where `localhost:9200` is the IP and port where Elasticsearch is listening.

=== Setting Up Filebeat to Use Logstash

If you want to use Logstash to perform additional processing on the data collected by
Filebeat, you need to set up Filebeat to use Logstash. For detailed steps, see
If you want to use Logstash to perform additional processing on the data collected by
Filebeat, you need to set up Filebeat to use Logstash. For detailed steps, see
{libbeat}/getting-started.html#logstash-setup[Setting Up Logstash].


Expand Down Expand Up @@ -184,6 +183,6 @@ mac:
sudo ./filebeat -e -c filebeat.yml -d "publish"
----------------------------------------------------------------------

Filebeat is now ready to send log files to your defined output.
Filebeat is now ready to send log files to your defined output.

Enjoy!
4 changes: 2 additions & 2 deletions docs/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ include::./overview.asciidoc[]

include::./getting-started.asciidoc[]

include::./fields.asciidoc[]

include::./configuration.asciidoc[]

include::./command-line.asciidoc[]

include::./fields.asciidoc[]

include::./migration.asciidoc[]

include::./support.asciidoc[]
43 changes: 25 additions & 18 deletions docs/migration.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ conversion to create a Filebeat config file for each Logstash Forwarder config f
Before migrating your config files, we recommend that you first read the <<filebeat-configuration-details>>
section to understand the Filebeat options.

NOTE: Logstash Forwarder has the option of auto completing environment variables in
NOTE: Logstash Forwarder has the option of autocompleting environment variables in
the configuration file. This option currently doesn't exist in Filebeat.

==== Migrating the "files" Section
Expand Down Expand Up @@ -341,27 +341,29 @@ The `config_dir` option specifies the path to the directory that contains additi

|`-cpuProfileFile`
|
|`cpuProfileFile` option was removed. You can use the profiling options of libbeat instead. For more details on profiling, see https://github.com/elastic/libbeat/issues/122.
|`cpuProfileFile` was removed. You can use the profiling options of libbeat instead. For more details on profiling, see https://github.com/elastic/libbeat/issues/122.

|`-quiet`
|
|The `quiet` option was removed. Libbeat is now used for logging, so you must use the libbeat {libbeat}/configuration.html#configuration-logging[logging options] instead.
|`quiet` was removed. Libbeat is now used for logging, so you must use the libbeat {libbeat}/configuration.html#configuration-logging[logging options] instead.


|===


=== Changes in the output fields
=== Changes to the Output Fields

In the default configuration, Filebeat structures its output documents a little
different from the Logstash Forwarder. This section discusses the differences
differently from the Logstash Forwarder. This section discusses the differences
and the options you have in case you want compatibility with the Logstash
Forwarder.

One such difference is that the custom fields (added from the configuration
file) are set top level in Logstash Forwarder but are grouped together under a
`fields` dictionary in Filebeat. If you need the old behavior during the
migration phase, you can use the <<fields-under-root>> configuration option:
==== Custom Fields are Grouped Under a "fields" Dictionary

The custom fields (added from the configuration file) are set as top-level
fields in Logstash Forwarder but are grouped together under a `fields`
dictionary in Filebeat. If you need the old behavior during the migration phase,
you can use the <<fields-under-root>> configuration option:

[source,yaml]
-------------------------------------------------------------------------------------
Expand All @@ -376,14 +378,17 @@ filebeat:
fields_under_root: true
-------------------------------------------------------------------------------------

Another difference is that while the Logstash Forwarder sends the hostname of
the server it is running on in the `host` field, Filebeat uses the
`beat.hostname` field for the same purpose. Because `host` is commonly used in
the Logstash plugin ecosystem, the Beats input plugin automatically copies
`beat.hostname` into `host`.
==== Filebeat uses "beat.hostname" for Sending the Hostname of the Server

While the Logstash Forwarder sends the hostname of the server it's running on in
the `host` field, Filebeat uses the `beat.hostname` field for the same purpose.
Because `host` is commonly used in the Logstash plugin ecosystem, the Beats
input plugin automatically copies `beat.hostname` into `host`.

Similarly, the `file` field was renamed to `source`. If you rely on this field
being named `file` you can rename it by using the mutate filter in Logstash. For
==== The "file" Field Was Renamed to "source"

The `file` field was renamed to `source`. If you rely on this field being
named `file`, you can rename it by using the mutate filter in Logstash. For
example:

[source,plain]
Expand All @@ -397,8 +402,10 @@ filter {
}
-------------------------------------------------------------------------------------

Finally, the `line` field was removed. This field wasn't correct in case of
restarts and making it correct would require a performance penalty. We recommend
==== The "line" Field Was Removed

The `line` field was removed. This field wasn't correct after restarts, and
making it correct would have resulted in a performance penalty. We recommend
using the `offset` field instead.

=== Other Changes
Expand Down
7 changes: 3 additions & 4 deletions docs/overview.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,13 @@


Filebeat is a log data shipper initially based on the https://github.com/elastic/logstash-forwarder[Logstash-Forwarder]
source code. Installed as an agent on your servers, Filebeat monitors the log directories or specific log files, tails the files,
source code. Installed as an agent on your servers, Filebeat monitors the log directories or specific log files, tails the files,
and forwards them either to https://www.elastic.co/products/logstash[Logstash] for parsing or directly to
https://www.elastic.co/products/elasticsearch[Elasticsearch] for indexing.

Here's how Filebeat works: When you start Filebeat, it starts one or more prospectors that look in the paths you've specified for log files. For each log file that the prospector locates, Filebeat starts a harvester. Each harvester reads a single log file for new content and sends the new log data to the spooler, which aggregates the events and sends the aggregated data to the output that you've configured for Filebeat.

image:./images/filebeat.png[Beats design]

Filebeat is a https://www.elastic.co/products/beats[Beat], and it is based on the libbeat framework.
The general configuration options for libbeat and how to setup Elasticsearch, Logstash, and Kibana are covered
https://www.elastic.co/guide/en/beats/libbeat/current/index.html[here].
Filebeat is a https://www.elastic.co/products/beats[Beat], and it is based on the libbeat framework.
The general configuration options for libbeat and how to setup Elasticsearch, Logstash, and Kibana are covered in the {libbeat}/index.html[Beats Platform Reference].

0 comments on commit 6557682

Please sign in to comment.