Merge pull request elastic#3554 from tsg/automatic_merge_from_master_…

…to_5.x_branch Automatic merge from master to 5.x branch
tsg · Feb 7, 2017 · 61ef81c · 61ef81c
2 parents 29644b1 + 0575fef
commit 61ef81c
Show file tree

Hide file tree

Showing 547 changed files with 11,965 additions and 3,263 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -74,10 +74,10 @@ matrix:
 
     # Generators
     - os: linux
-      env: TARGETS="-C generate/metricbeat/metricset test"
+      env: TARGETS="-C generator/metricbeat test"
       go: *go_version
     - os: linux
-      env: TARGETS="-C generate/beat test"
+      env: TARGETS="-C generator/beat test"
       go: *go_version
 
 addons:
@@ -88,6 +88,8 @@ addons:
       - geoip-database
 
 before_install:
+  - umask 022
+  - chmod -R go-w $GOPATH/src/github.com/elastic/beats
   # Docker-compose installation
   - sudo rm /usr/local/bin/docker-compose || true
   - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose

diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
@@ -14,107 +14,183 @@ https://github.com/elastic/beats/compare/v5.1.1...master[Check the HEAD diff]
 
 *Affecting all Beats*
 
+- Change beat generator. Use `$GOPATH/src/github.com/elastic/beats/script/generate.py` to generate a beat. {pull}3452[3452]
+- Configuration files must not be writable by other users. {pull}3544[3544]
+
+*Filebeat*
+- Always use absolute path for event and registry. This can lead to issues when relative paths were used before. {pull}3328[3328]
+
+*Heartbeat*
+
 *Metricbeat*
+- Linux cgroup metrics are now enabled by default for the system process
+  metricset. The configuration option for the feature was renamed from
+  `cgroups` to `process.cgroups.enabled`. {pull}3519[3519]
+- Change fieldnames couchbase.node.couch.*.actual_disk_size.* to couchbase.node.couch.*.disk_size.* {pull}3545[3545]
 
 *Packetbeat*
 
-*Topbeat*
+*Winlogbeat*
+
+
+==== Bugfixes
+
+*Affecting all Beats*
+
+- Add `_id`, `_type`, `_index` and `_score` fields in the generated index pattern. {pull}3282[3282]
 
 *Filebeat*
+- Always use absolute path for event and registry. {pull}3328[3328]
+
+*Heartbeat*
+
+*Metricbeat*
+
+- Fix go routine leak in docker module. {pull}3492[3492]
+
+*Packetbeat*
 
 *Winlogbeat*
 
 
-==== Bugfixes
+==== Added
 
 *Affecting all Beats*
 
-- Add `_id`, `_type`, `_index` and `_score` fields in the generated index pattern. {pull}3282}[3282]
+- Files created by Beats (logs, registry, file output) will have 0600 permissions. {pull}3387[3387].
+- RPM/deb packages will now install the config file with 0600 permissions. {pull}3382[3382]
+- Add the option to pass custom HTTP headers to the Elasticsearch output. {pull}3400[3400]
+- Unify `regexp` and `contains` conditionals, for both to support array of strings and convert numbers to strings if required. {pull}3469[3469]
+- Add the option to load the sample dashboards during the Beat startup phase. {pull}3506[3506]
+- Disabled date detection in Elasticsearch index templates. Date fields must be explicitly defined in index templates. {pull}3528[3528]
+- Using environment variables in the configuration file is now GA, instead of experimental. {pull}3525[3525]
+
+*Filebeat*
+
+- Add the `pipeline` config option at the prospector level, for configuring the Ingest Node pipeline ID. {pull}3433[3433]
+- Update regular expressions used for matching file names or lines (multiline, include/exclude functionality) to new matchers improving performance of simple string matches. {pull}3469[3469]
+- The `symlinks` and `harverster_limit` settings are now GA, instead of experimental. {pull}3525[3525]
+- close_timeout is also applied when the output is blocking. {pull}3511[3511]
+
+*Heartbeat*
 
 *Metricbeat*
 
-- Fix service times-out at startup. {pull}3056[3056]
-- Kafka module case sensitive host name matching. {pull}3193[3193]
-- Fix interface conversion panic in couchbase module {pull}3272[3272]
-- Fix overwriting explicit empty config sections {issue}2918[2918]
+- Add experimental dbstats metricset to MongoDB module. {pull}3228[3228]
+- Use persistent, direct connections to the configured nodes for MongoDB module. {pull}3228[3228]
+- Add dynamic configuration reloading for modules. {pull}3281[3281]
+- Add docker health metricset {pull}3357[3357]
+- Add docker image metricset {pull}3467[3467]
+- System module uses new matchers for white-listing processes. {pull}3469[3469]
+- Add Beta CEPH module with health metricset. {pull}3311[3311]
+- Add Beta php_fpm module with pool metricset. {pull}3415[3415]
+- The Docker, Kafka, and Prometheus modules are now Beta, instead of experimental. {pull}3525[3525]
+- The HAProxy module is now GA, instead of experimental. {pull}3525[3525]
 
 *Packetbeat*
 
-- Fix issue where some Cassandra visualizations were showing data from all protocols. {issue}3314[3314]
+*Winlogbeat*
 
-*Topbeat*
+==== Deprecated
+
+*Affecting all Beats*
+
+- Usage of field _type is deprecated. It should not be used in queries or dashboards. {pull}3409[3409]
 
 *Filebeat*
-- Fix registry cleanup issue when files falling under ignore_older after restart. {issue}2818[2818]
-- Fix registry migration issue from old states were files were only harvested after second restart. {pull}3322[3322]
-- Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing on 32 bits system. {issue}3273[3273]
+
+- The experimental `publish_async` option is now deprecated and is planned to be removed in 6.0. {pull}3525[3525]
+
+*Heartbeat*
+
+*Metricbeat*
+
+*Packetbeat*
 
 *Winlogbeat*
-- Fix for "The array bounds are invalid" error when reading large events. {issue}3076[3076]
+
+==== Knwon Issue
+
+*Filebeat*
+- Prospector reloading only works properly with new files. {pull}3546[3546]
+
+////////////////////////////////////////////////////////////
+
+[[release-notes-5.2.0]]
+=== Beats version 5.2.0
+https://github.com/elastic/beats/compare/v5.1.2...v5.2.0[View commits]
+
+==== Bugfixes
+
+*Affecting all Beats*
+
+- Fix overwriting explicit empty config sections. {issue}2918[2918]
+
+*Filebeat*
+
+- Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing on 32 bits system. {issue}3273[3273]
+
+*Metricbeat*
+
+- Fix service times-out at startup. {pull}3056[3056]
+- Kafka module case sensitive host name matching. {pull}3193[3193]
+- Fix interface conversion panic in couchbase module {pull}3272[3272]
+
+*Packetbeat*
+
+- Fix issue where some Cassandra visualizations were showing data from all protocols. {issue}3314[3314]
 
 ==== Added
 
 *Affecting all Beats*
+
 - Add support for passing list and dictionary settings via -E flag.
 - Support for parsing list and dictionary setting from environment variables.
 - Added new flags to import_dashboards (-cacert, -cert, -key, -insecure). {pull}3139[3139] {pull}3163[3163]
 - The limit for the number of fields is increased via the mapping template. {pull}3275[3275]
 - Updated to Go 1.7.4. {pull}3277[3277]
 - Added a NOTICE file containing the notices and licenses of the dependencies. {pull}3334[3334].
-- Files created by Beats (logs, registry, file output) will have 0600 permissions. {pull}3387[3387].
-- RPM/deb packages will now install the config file with 0600 permissions. {pull}3382[3382]
+
+*Heartbeat*
+
+- First release, containing monitors for ICMP, TCP, and HTTP.
+
+*Filebeat*
+
+- Add enabled config option to prospectors. {pull}3157[3157]
+- Add target option for decoded_json_field. {pull}3169[3169]
 
 *Metricbeat*
 
-- Add experimental filebeat metricset in the beats module. {pull}2297[2297]
-- Add experimental libbeat metricset in the beats module. {pull}2339[2339]
-- Add experimental docker module. Provided by Ingensi and @douaejeouit based on dockbeat.
-- Add username and password config options to the MongoDB module. {pull}2889[2889]
-- Add username and password config options to the PostgreSQL module. {pull}2889[2890]
-- Add system core metricset for Windows. {pull}2883[2883]
-- Add a sample Redis Kibana dashboard. {pull}2916[2916]
-- Add support for MongoDB 3.4 and WiredTiger metrics. {pull}2999[2999]
-- Add experimental kafka module with partition metricset. {pull}2969[2969]
-- Add raw config option for mysql/status metricset. {pull}3001[3001]
-- Add experimental dbstats metricset to MongoDB module. {pull}3228[3228]
-- Use persistent, direct connections to the configured nodes for MongoDB module. {pull}3228[3228]
 - Kafka module broker matching enhancements. {pull}3129[3129]
 - Add a couchbase module with metricsets for node, cluster and bucket. {pull}3081[3081]
-- Export number of cores for cpu module. {pull}3192[3192]
+- Export number of cores for CPU module. {pull}3192[3192]
 - Experimental Prometheus module. {pull}3202[3202]
 - Add system socket module that reports all TCP sockets. {pull}3246[3246]
 - Kafka consumer groups metricset. {pull}3240[3240]
-- Add dynamic configuration reloading for modules. {pull}3281[3281]
-
-*Packetbeat*
-
-*Topbeat*
-
-*Filebeat*
-- Add enabled config option to prospectors. {pull}3157[3157]
-- Add target option for decoded_json_field. {pull}3169[3169]
 
 *Winlogbeat*
 
-- Reduced amount of memory allocated while reading event log records. {pull}3113[3113] {pull}3118[3113]
+- Reduced amount of memory allocated while reading event log records. {pull}3113[3113] {pull}3118[3118]
 
-==== Deprecated
+[[release-notes-5.1.2]]
+=== Beats version 5.1.2
+https://github.com/elastic/beats/compare/v5.1.1...v5.1.2[View commits]
 
-*Affecting all Beats*
-- Usage of field _type is deprecated. It should not be used in queries or dashboards. {pull}3409[3409]
+==== Bugfixes
 
-*Metricbeat*
+*Filebeat*
+
+- Fix registry migration issue from old states where files were only harvested after second restart. {pull}3322[3322]
 
 *Packetbeat*
 
-*Topbeat*
-- Fix error on importing dashboards due to colons in the Caassandra dashboard. {issue}3140[3140]
-
-*Filebeat*
+- Fix error on importing dashboards due to colons in the Cassandra dashboard. {issue}3140[3140]
+- Fix error on importing dashboards due to the wrong type for the geo_point fields. {pull}3147[3147]
 
 *Winlogbeat*
 
-////////////////////////////////////////////////////////////
+- Fix for "The array bounds are invalid" error when reading large events. {issue}3076[3076]
 
 [[release-notes-5.1.1]]
 === Beats version 5.1.1

diff --git a/Makefile b/Makefile
@@ -11,7 +11,7 @@ SNAPSHOT?=yes
 .PHONY: testsuite
 testsuite:
 	$(foreach var,$(PROJECTS),$(MAKE) -C $(var) testsuite || exit 1;)
-	#$(MAKE) -C generate test
+	#$(MAKE) -C generator test
 
 stop-environments:
 	$(foreach var,$(PROJECTS_ENV),$(MAKE) -C $(var) stop-environment || exit 0;)
@@ -46,18 +46,18 @@ update:
 clean:
 	rm -rf build
 	$(foreach var,$(PROJECTS),$(MAKE) -C $(var) clean || exit 1;)
-	$(MAKE) -C generate clean
+	$(MAKE) -C generator clean
 
 # Cleans up the vendor directory from unnecessary files
 # This should always be run after updating the dependencies
 .PHONY: clean-vendor
 clean-vendor:
-	sh scripts/clean_vendor.sh
+	sh script/clean_vendor.sh
 
 .PHONY: check
 check:
 	$(foreach var,$(PROJECTS),$(MAKE) -C $(var) check || exit 1;)
-	# Validate that all updates were commited
+	# Validate that all updates were committed
 	$(MAKE) update
 	git update-index --refresh
 	git diff-index --exit-code HEAD --
@@ -118,5 +118,3 @@ upload-release:
 .PHONY: notice
 notice:
 	python dev-tools/generate_notice.py .
-
-
diff --git a/README.md b/README.md
@@ -21,6 +21,7 @@ framework for creating Beats, and all the officially supported Beats:
 Beat  | Description
 --- | ---
 [Filebeat](https://github.com/elastic/beats/tree/master/filebeat) | Tails and ships log files
+[Heartbeat](https://github.com/elastic/beats/tree/master/heartbeat) | Ping remote services for availability
 [Metricbeat](https://github.com/elastic/beats/tree/master/metricbeat) | Fetches sets of metrics from the operating system and services
 [Packetbeat](https://github.com/elastic/beats/tree/master/packetbeat) | Monitors the network and applications by sniffing packets
 [Winlogbeat](https://github.com/elastic/beats/tree/master/winlogbeat) | Fetches and ships Windows Event logs
@@ -38,6 +39,7 @@ on the [elastic.co site](https://www.elastic.co/guide/):
 
 * [Beats platform](https://www.elastic.co/guide/en/beats/libbeat/current/index.html)
 * [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/index.html)
+* [Heartbeat](https://www.elastic.co/guide/en/beats/heartbeat/current/index.html)
 * [Metricbeat](https://www.elastic.co/guide/en/beats/metricbeat/current/index.html)
 * [Packetbeat](https://www.elastic.co/guide/en/beats/packetbeat/current/index.html)
 * [Winlogbeat](https://www.elastic.co/guide/en/beats/winlogbeat/current/index.html)

diff --git a/Vagrantfile b/Vagrantfile
@@ -9,7 +9,7 @@
 # This box is used as a Windows development and testing environment for Beats.
 #
 # Usage and Features:
-#   - Two users exist: Administartor and Vagrant. Both have the password: vagrant
+#   - Two users exist: Administrator and Vagrant. Both have the password: vagrant
 #   - Use 'vagrant ssh' to open a Windows command prompt.
 #   - Use 'vagrant rdp' to open a Windows Remote Deskop session. Mac users must
 #     install the Microsoft Remote Desktop Client from the App Store.

diff --git a/dev-tools/README.md b/dev-tools/README.md
@@ -44,7 +44,7 @@ pip install -r requirements.txt
 
 This creates the environment that contains all the python packages required to
 run the `export_dashboards.py` script. Thus, for the next runs you just need
-to enable the enviroment:
+to enable the environment:
 
 ```
 . env/bin/activate

diff --git a/dev-tools/aggregate_coverage.py b/dev-tools/aggregate_coverage.py
@@ -8,6 +8,7 @@
 import argparse
 import fnmatch
 
+
 def main(arguments):
 
     parser = argparse.ArgumentParser(description=__doc__,
@@ -34,7 +35,7 @@ def main(arguments):
                     if not line.startswith('mode:') and "vendor" not in line:
                         (position, stmt, count) = line.split(" ")
                         stmt = int(stmt)
-                        count = int (count)
+                        count = int(count)
                         prev_count = 0
                         if lines.has_key(position):
                             (_, prev_stmt, prev_count) = lines[position]

diff --git a/dev-tools/generate_notice.py b/dev-tools/generate_notice.py
@@ -49,7 +49,6 @@ def add_licenses(f, licenses):
                     f.write(read_file(notice_file))
 
 
-
 def create_notice(filename, beat, copyright, licenses):
 
     now = datetime.datetime.now()

diff --git a/dev-tools/jenkins_ci b/dev-tools/jenkins_ci
@@ -131,6 +131,7 @@ main() {
     err "--build and --cleanup cannot be used together"
     exit 1
   elif [ "$BUILD" == "true" ]; then
+    chmod -R go-w "${GOPATH}/src/github.com/elastic/beats"
     build
   elif [ "$CLEANUP" == "true" ]; then
     cleanup
@@ -140,4 +141,5 @@ main() {
   fi
 }
 
+umask 022
 main $*
diff --git a/filebeat/.gitignore b/filebeat/.gitignore
@@ -6,4 +6,7 @@ filebeat
 
 build
 _meta/kibana
+_meta/module.generated
+_meta/beat.yml
+_meta/beat.full.yml
 /tests/load/logs
diff --git a/filebeat/Makefile b/filebeat/Makefile
@@ -27,10 +27,26 @@ fields:
 	cat ${ES_BEATS}/filebeat/_meta/fields.common.yml > _meta/fields.generated.yml
 	. ${PYTHON_ENV}/bin/activate; python ${ES_BEATS}/metricbeat/scripts/fields_collector.py >> _meta/fields.generated.yml
 
+# Collects all modules files to be packaged in a temporary folder
+.PHONY: modules
+modules:
+	mkdir -p _meta/
+	rm -rf _meta/module.generated
+	rsync -av module/ _meta/module.generated --exclude "_meta" --exclude "*/*/test"
+
+# Collects all module configs
+.PHONY: configs
+configs: python-env
+	cat ${ES_BEATS}/filebeat/_meta/common.p1.yml > _meta/beat.yml
+	. ${PYTHON_ENV}/bin/activate; python ${ES_BEATS}/metricbeat/scripts/config_collector.py --beat ${BEAT_NAME} $(PWD) >> _meta/beat.yml
+	cat ${ES_BEATS}/filebeat/_meta/common.p2.yml >> _meta/beat.yml
+	cat ${ES_BEATS}/filebeat/_meta/common.full.p1.yml > _meta/beat.full.yml
+	. ${PYTHON_ENV}/bin/activate; python ${ES_BEATS}/metricbeat/scripts/config_collector.py --beat ${BEAT_NAME} --full $(PWD) >> _meta/beat.full.yml
+	cat ${ES_BEATS}/filebeat/_meta/common.full.p2.yml >> _meta/beat.full.yml
 
 # Runs all collection steps and updates afterwards
 .PHONY: collect
-collect: fields kibana
+collect: fields kibana modules configs
 
 
 # Creates a new fileset. Requires the params MODULE and FILESET
Original file line number	Diff line number	Diff line change
Expand Up		@@ -49,7 +49,6 @@ def add_licenses(f, licenses):
		f.write(read_file(notice_file))



		def create_notice(filename, beat, copyright, licenses):

		now = datetime.datetime.now()
Expand Down