Skip to content

Commit

Permalink
Merge pull request elastic#3813 from tsg/automatic_merge_from_5.3_to_…
Browse files Browse the repository at this point in the history
…5.x_branch

Automatic merge from 5.3 to 5.x branch
  • Loading branch information
ruflin authored Mar 27, 2017
2 parents 1b778ef + e7b40e5 commit c71bbb6
Show file tree
Hide file tree
Showing 106 changed files with 2,268 additions and 415 deletions.
133 changes: 107 additions & 26 deletions CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,50 +8,122 @@
// Template, add newest changes here

=== Beats version HEAD
https://github.com/elastic/beats/compare/v5.1.1...master[Check the HEAD diff]
https://github.com/elastic/beats/compare/v5.3.0...master[Check the HEAD diff]

==== Breaking changes

*Affecting all Beats*

- Change beat generator. Use `$GOPATH/src/github.com/elastic/beats/script/generate.py` to generate a beat. {pull}3452[3452]
- Configuration files must not be writable by other users. {pull}3544[3544]
*Filebeat*

*Heartbeat*

*Metricbeat*

*Packetbeat*

*Winlogbeat*

==== Bugfixes

*Affecting all Beats*


*Filebeat*
- Always use absolute path for event and registry. This can lead to issues when relative paths were used before. {pull}3328[3328]

*Heartbeat*

*Metricbeat*
- Linux cgroup metrics are now enabled by default for the system process
metricset. The configuration option for the feature was renamed from
`cgroups` to `process.cgroups.enabled`. {pull}3519[3519]
- Change fieldnames couchbase.node.couch.*.actual_disk_size.* to couchbase.node.couch.*.disk_size.* {pull}3545[3545]

*Packetbeat*

*Winlogbeat*


==== Bugfixes
==== Added

*Affecting all Beats*

- Add `_id`, `_type`, `_index` and `_score` fields in the generated index pattern. {pull}3282[3282]
- Update index mappings to support future Elasticsearch 6.X. {pull}3778[3778]

*Filebeat*
- Always use absolute path for event and registry. {pull}3328[3328]

*Heartbeat*

*Metricbeat*

- Fix go routine leak in docker module. {pull}3492[3492]
*Packetbeat*

*Winlogbeat*

==== Deprecated

*Affecting all Beats*

*Filebeat*

*Heartbeat*

*Metricbeat*

*Packetbeat*

*Winlogbeat*

==== Knwon Issue

*Affecting all Beats*

*Filebeat*

*Heartbeat*

*Metricbeat*

*Packetbeat*

*Winlogbeat*

////////////////////////////////////////////////////////////
[[release-notes-5.3.0]]
=== Beats version 5.3.0
https://github.com/elastic/beats/compare/v5.2.2...v5.3.0[View commits]
==== Breaking changes
*Affecting all Beats*
- Configuration files must be owned by the user running the Beat or by root, and they must not be writable by others. {pull}3544[3544] {pull}3689[3689]
- Change Beat generator. Use `$GOPATH/src/github.com/elastic/beats/script/generate.py` to generate a beat. {pull}3452[3452]
*Filebeat*
- Always use absolute path for event and registry. This can lead to issues when relative paths were used before. {pull}3328[3328]
*Metricbeat*
- Linux cgroup metrics are now enabled by default for the system process metricset. The configuration option for the feature was renamed from `cgroups` to `process.cgroups.enabled`. {pull}3519[3519]
- Change field names `couchbase.node.couch.*.actual_disk_size.*` to `couchbase.node.couch.*.disk_size.*` {pull}3545[3545]
==== Bugfixes
*Affecting all Beats*
- Add `_id`, `_type`, `_index` and `_score` fields in the generated index pattern. {pull}3282[3282]
*Filebeat*
- Fix empty registry file on machine crash. {issue}3537[3537]
*Metricbeat*
- Add error handling to system process metricset for when Linux cgroups are missing from the kernel. {pull}3692[3692]
- Add labels to the Docker healthcheck metricset output. {pull}3707[3707]
*Winlogbeat*
- Fix handling of empty strings in event_data. {pull}3705[3705]
==== Added
Expand All @@ -64,16 +136,16 @@ https://github.com/elastic/beats/compare/v5.1.1...master[Check the HEAD diff]
- Add the option to load the sample dashboards during the Beat startup phase. {pull}3506[3506]
- Disabled date detection in Elasticsearch index templates. Date fields must be explicitly defined in index templates. {pull}3528[3528]
- Using environment variables in the configuration file is now GA, instead of experimental. {pull}3525[3525]
- Update index mappings to support future Elasticsearch 6.X. {pull}3778[3778]
*Filebeat*
- Add Filebeat modules for system, apache2, mysql, and nginx. {issue}3159[3159]
- Add the `pipeline` config option at the prospector level, for configuring the Ingest Node pipeline ID. {pull}3433[3433]
- Update regular expressions used for matching file names or lines (multiline, include/exclude functionality) to new matchers improving performance of simple string matches. {pull}3469[3469]
- The `symlinks` and `harverster_limit` settings are now GA, instead of experimental. {pull}3525[3525]
- close_timeout is also applied when the output is blocking. {pull}3511[3511]
- Improve handling of different path variants on Windows. {pull}3781[3781]
*Heartbeat*
*Metricbeat*
Expand All @@ -87,35 +159,44 @@ https://github.com/elastic/beats/compare/v5.1.1...master[Check the HEAD diff]
- Add Beta php_fpm module with pool metricset. {pull}3415[3415]
- The Docker, Kafka, and Prometheus modules are now Beta, instead of experimental. {pull}3525[3525]
- The HAProxy module is now GA, instead of experimental. {pull}3525[3525]

*Packetbeat*

*Winlogbeat*
- Add the ability to collect the environment variables from system processes. {pull}3337[3337]
==== Deprecated
*Affecting all Beats*
- Usage of field _type is deprecated. It should not be used in queries or dashboards. {pull}3409[3409]
- Usage of field `_type` is deprecated. It should not be used in queries or dashboards. {pull}3409[3409]
*Filebeat*
- The experimental `publish_async` option is now deprecated and is planned to be removed in 6.0. {pull}3525[3525]
*Heartbeat*
[[release-notes-5.2.2]]
=== Beats version 5.2.2
https://github.com/elastic/beats/compare/v5.2.1...v5.2.2[View commits]
*Metricbeat*
*Packetbeat*
- Fix bug docker module hanging when docker container killed. {issue}3610[3610]
- Set timeout to period instead of 1s by default as documented. {pull}3612[3612]
*Winlogbeat*
[[release-notes-5.2.1]]
=== Beats version 5.2.1
https://github.com/elastic/beats/compare/v5.2.0...v5.2.1[View commits]
==== Knwon Issue
==== Bugfixes
*Filebeat*
- Prospector reloading only works properly with new files. {pull}3546[3546]
*Metricbeat*
////////////////////////////////////////////////////////////
- Fix go routine leak in docker module. {pull}3492[3492]
*Packetbeat*
- Fix error in the NFS sample dashboard. {pull}3548[3548]
*Winlogbeat*
- Fix error in the Winlogbeat sample dashboard. {pull}3548[3548]
[[release-notes-5.2.0]]
=== Beats version 5.2.0
Expand Down
23 changes: 21 additions & 2 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -1505,10 +1505,29 @@ For more information, please see http://www.bis.doc.gov
See also http://www.apache.org/dev/crypto.html and/or seek legal counsel.

--------------------------------------------------------------------
github.com/opencontainers/runtime-spec
github.com/Sirupsen/logrus
--------------------------------------------------------------------
Apache License
The MIT License (MIT)

Copyright (c) 2014 Simon Eskildsen

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

--------------------------------------------------------------------
github.com/go-sql-driver/mysql
Expand Down
12 changes: 10 additions & 2 deletions filebeat/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,23 @@ modules:
.PHONY: configs
configs: python-env
cat ${ES_BEATS}/filebeat/_meta/common.p1.yml > _meta/beat.yml
. ${PYTHON_ENV}/bin/activate; python ${ES_BEATS}/metricbeat/scripts/config_collector.py --beat ${BEAT_NAME} $(PWD) >> _meta/beat.yml
# disabled wile filebeat modules are Beta
#. ${PYTHON_ENV}/bin/activate; python ${ES_BEATS}/metricbeat/scripts/config_collector.py --beat ${BEAT_NAME} $(PWD) >> _meta/beat.yml
cat ${ES_BEATS}/filebeat/_meta/common.p2.yml >> _meta/beat.yml
cat ${ES_BEATS}/filebeat/_meta/common.full.p1.yml > _meta/beat.full.yml
. ${PYTHON_ENV}/bin/activate; python ${ES_BEATS}/metricbeat/scripts/config_collector.py --beat ${BEAT_NAME} --full $(PWD) >> _meta/beat.full.yml
cat ${ES_BEATS}/filebeat/_meta/common.full.p2.yml >> _meta/beat.full.yml

# Collects all module docs
.PHONY: collect-docs
collect-docs: python-env
-rm -rf docs/modules
mkdir -p docs/modules
. ${PYTHON_ENV}/bin/activate; python ${ES_BEATS}/filebeat/scripts/docs_collector.py --beat ${BEAT_NAME}

# Runs all collection steps and updates afterwards
.PHONY: collect
collect: fields kibana modules configs
collect: fields kibana modules configs collect-docs


# Creates a new fileset. Requires the params MODULE and FILESET
Expand Down
3 changes: 0 additions & 3 deletions filebeat/_meta/common.p2.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
# For more available modules and options, please see the filebeat.full.yml sample
# configuration file.

#=========================== Filebeat prospectors =============================

filebeat.prospectors:
Expand Down
2 changes: 1 addition & 1 deletion filebeat/beater/filebeat.go
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ func (fb *Filebeat) Run(b *beat.Beat) error {
return err
}

crawler, err := crawler.New(newSpoolerOutlet(fb.done, spooler, wgEvents), config.Prospectors, *once)
crawler, err := crawler.New(newSpoolerOutlet(fb.done, spooler, wgEvents), config.Prospectors, fb.done, *once)
if err != nil {
logp.Err("Could not init crawler: %v", err)
return err
Expand Down
90 changes: 90 additions & 0 deletions filebeat/channel/outlet.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
package channel

import (
"sync"
"sync/atomic"

"github.com/elastic/beats/filebeat/input"
)

// Outlet struct is used to be passed to an object which needs an outlet
//
// The difference between signal and done channel is as following:
// - signal channel can be added through SetSignal and is used to
// interrupt events sent through OnEventSignal-
// - done channel is used to close and stop the outlet
//
// If SetSignal is used, it must be ensure that there is only one event producer.
type Outlet struct {
wg *sync.WaitGroup // Use for counting active events
done <-chan struct{}
signal <-chan struct{}
channel chan *input.Event
isOpen int32 // atomic indicator
}

func NewOutlet(
done <-chan struct{},
c chan *input.Event,
wg *sync.WaitGroup,
) *Outlet {
return &Outlet{
done: done,
channel: c,
wg: wg,
isOpen: 1,
}
}

// SetSignal sets the signal channel for OnEventSignal
// If SetSignal is used, it must be ensure that only one producer exists.
func (o *Outlet) SetSignal(signal <-chan struct{}) {
o.signal = signal
}

func (o *Outlet) OnEvent(event *input.Event) bool {
open := atomic.LoadInt32(&o.isOpen) == 1
if !open {
return false
}

if o.wg != nil {
o.wg.Add(1)
}

select {
case <-o.done:
if o.wg != nil {
o.wg.Done()
}
atomic.StoreInt32(&o.isOpen, 0)
return false
case o.channel <- event:
return true
}
}

// OnEventSignal can be stopped by the signal that is set with SetSignal
// This does not close the outlet. Only OnEvent does close the outlet.
// If OnEventSignal is used, it must be ensured that only one producer is used.
func (o *Outlet) OnEventSignal(event *input.Event) bool {
open := atomic.LoadInt32(&o.isOpen) == 1
if !open {
return false
}

if o.wg != nil {
o.wg.Add(1)
}

select {
case <-o.signal:
if o.wg != nil {
o.wg.Done()
}
o.signal = nil
return false
case o.channel <- event:
return true
}
}
Loading

0 comments on commit c71bbb6

Please sign in to comment.