Merge branch 'master' into apm-server-outputs

CHANGELOG.asciidoc

@@ -3,6 +3,33 @@
 :issue: https://github.com/elastic/beats/issues/
 :pull: https://github.com/elastic/beats/pull/
 
+[[release-notes-8.0.0-alpha1]]
+=== Beats version 8.0.0-alpha1
+
+Changes will be described in a later alpha / beta.
+
+
+[[release-notes-7.13.4]]
+=== Beats version 7.13.4
+https://github.com/elastic/beats/compare/v7.13.3...v7.13.4[View commits]
+
+==== Bugfixes
+
+*Affecting all Beats*
+
+- Fix `add_process_metadata` processor complaining about valid pid fields not being valid integers. {pull}26829[26829] {issue}26830[26830]
+
+*Auditbeat*
+
+- Do not close filestream harvester if an unexpected error is returned when `close.on_state_change.*` is enabled. {pull}26411[26411]
+
+*Filebeat*
+
+- Fix Elasticsearch compatibility for modules that use `type: ip` with `convert` processors. {issue}26629[26629] {pull}26676[26676]
+- Fix Elasticsearch compatibility for modules that use the `network_direction` processor. {issue}26629[26629] {pull}26676[26676]
+- Fix Elasticsearch compatibility for modules that use the `registered_domain` processor. {issue}26629[26629] {pull}26676[26676]
+
+
 [[release-notes-7.13.3]]
 === Beats version 7.13.3
 https://github.com/elastic/beats/compare/v7.13.2...v7.13.3[View commits]

CHANGELOG.next.asciidoc

@@ -35,6 +35,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Removed beats central management {pull}25696[25696], {issue}23908[23908]
 - MacOSX minimum supported version set to 10.14 {issue}24193{24193}
 - Add daemonset.name in pods controlled by DaemonSets {pull}26808[26808], {issue}25816[25816]
+- Kubernetes autodiscover fails in node scope if node name cannot be discovered {pull}26947[26947]
 
 *Auditbeat*
 
@@ -140,6 +141,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Added `statsd.mappings` configuration for Statsd module {pull}26220[26220]
 - Added Airflow lightweight module {pull}26220[26220]
 - Add state_job metricset to Kubernetes module{pull}26479[26479]
+- Recover service.address field in vsphere module {issue}26902[26902] {pull}26904[26904]
 
 *Packetbeat*
 
@@ -244,7 +246,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Omit full index template from errors that occur while loading the template. {pull}25743[25743]
 - In the script processor, the `decode_xml` and `decode_xml_wineventlog` processors are now available as `DecodeXML` and `DecodeXMLWineventlog` respectively.
 - Fix encoding errors when using the disk queue on nested data with multi-byte characters {pull}26484[26484]
-- Fix `add_process_metadata` processor complaining about valid pid fields not being valid integers. {pull}26829[26829] {issue}26830[26830]
+- Preserve annotations in a kubernetes namespace metadata {pull}27045[27045]
 
 *Auditbeat*
 
@@ -283,7 +285,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix CredentialsJSON unpacking for `gcp-pubsub` and `httpjson` inputs. {pull}23277[23277]
 - Fix issue with m365_defender, when parsing incidents that has no alerts attached: {pull}25421[25421]
 - Fix default config template values for paths on oracle module: {pull}26276[26276]
-- Do not close filestream harvester if an unexpected error is returned when close.on_state_change.* is enabled. {pull}26411[26411]
 - Fix Elasticsearch compatibility for modules that use `copy_from` in `set` processors. {issue}26629[26629]
 - Change type of max_bytes in all configs to be cfgtype.ByteSize {pull}26699[26699]
 
@@ -395,11 +396,11 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix `kibana.log` pipeline when `event.duration` calculation becomes a Long. {issue}24556[24556] {pull}25675[25675]
 - Removed incorrect `http.request.referrer` field from `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
 - Fix `threatintel.indicator.url.full` not being populated. {issue}26351[26351] {pull}26508[26508]
-- Fix Elasticsearch compatibility for modules that use `type: ip` with `convert` processors. {issue}26629[26629] {pull}26676[26676]
-- Fix Elasticsearch compatibility for modules that use the `network_direction` processor. {issue}26629[26629] {pull}26676[26676]
-- Fix Elasticsearch compatibility for modules that use the `registered_domain` processor. {issue}26629[26629] {pull}26676[26676]
 - Fix Suricata metadata fields breaking visualizations, moved out of flattened datatype. {pull}26710[26710]
 - Fix `httpjson` template data key for `url.params`. {pull}26848[26848]
+- Cisco asa/ftd: Fix reversed usage of observer ingress and egress interfaces. {pull}26265[26265]
+- Fix `aws.s3access` pipeline when remote IP is a `-`. {issue}26913[26913] {pull}26940[26940]
+- Fix service name in aws-cloudwatch input from cloudwatchlogs to logs. {pull}27007[27007]
 
 *Heartbeat*
 
@@ -511,6 +512,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix GCP Project ID being ingested as `cloud.account.id` in `gcp.billing` module {issue}26357[26357] {pull}26412[26412]
 - Fix memory leak in SQL module when database is not available. {issue}25840[25840] {pull}26607[26607]
 - Fix aws metric tags with resourcegroupstaggingapi paginator.  {issue}26385[26385] {pull}26443[26443]
+- Fix quoting in GCP billing table name  {issue}26855[26855] {pull}26870[26870]
+- Allow metric prefix override per service in gcp module. {pull}26960[26960]
 
 *Packetbeat*
 
@@ -528,6 +531,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Functionbeat*
 
+- Expose region in AWS configuration so Functionbeat can deploy the Lamba in the correct place. {pull}26523[26523]
+
 *Elastic Logging Plugin*
 
 
@@ -612,6 +617,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add orchestrator.cluster.name/url fields as k8s metadata {pull}26056[26056]
 - Libbeat: report beat version to monitoring. {pull}26214[26214]
 - Ensure common proxy settings support in HTTP clients: proxy_disabled, proxy_url, proxy_headers and typical environment variables HTTP_PROXY, HTTPS_PROXY, NOPROXY. {pull}25219[25219]
+- `add_process_metadata` processor enrich process information with owner name and id. {issue}21068[21068] {pull}21111[21111]
+- Add proxy support for AWS functions. {pull}26832[26832]
+- Add sha256 digests to RPM packages. {issue}23670[23670]
 
 *Auditbeat*
 
@@ -857,6 +865,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Update `fortinet` ingest pipelines. {issue}22136[22136] {issue}25254[25254] {pull}24816[24816]
 - Use default add_locale for fortinet.firewall {issue}20300[20300] {pull}26524[26524]
 - Add new template functions and `value_type` parameter to `httpjson` transforms. {pull}26847[26847]
+- Add support to merge registry updates in the filestream input across multiple ACKed batches in case of backpressure in the registry or disk. {pull}25976[25976]
+- Update Elasticsearch module's ingest pipeline for parsing new deprecation logs {issue}26857[26857] {pull}26880[26880]
 
 *Heartbeat*
 
@@ -997,6 +1007,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add total CPU to vSphere virtual machine metrics. {pull}26167[26167]
 - Add AWS Kinesis metricset. {pull}25989[25989]
 - Move openmetrics module to oss. {pull}26561[26561]
+- Add Cluster filter on ECS Kubernetes overview dashboard and corresponding section on Kubernetes module documentation page. {pull}26919[26919]
 
 *Packetbeat*
 

Jenkinsfile

@@ -202,13 +202,7 @@ def runLinting() {
       mapParallelTasks["${k}"] = v
     }
   }
-  mapParallelTasks['default'] = {
-                                cmd(label: "make check-python", script: "make check-python")
-                                cmd(label: "make notice", script: "make notice")
-                                // `make check-go` must follow `make notice` to ensure that the lint checks can be satisfied
-                                cmd(label: "make check-go", script: "make check-go")
-                                cmd(label: "Check for changes", script: "make check-no-changes")
-                              }
+  mapParallelTasks['default'] = { cmd(label: 'make check-default', script: 'make check-default') }
 
   parallel(mapParallelTasks)
 }
@@ -399,10 +393,13 @@ def publishPackages(beatsFolder){
 * @param beatsFolder the beats folder.
 */
 def uploadPackages(bucketUri, beatsFolder){
-  googleStorageUploadExt(bucket: bucketUri,
-    credentialsId: "${JOB_GCS_EXT_CREDENTIALS}",
-    pattern: "${beatsFolder}/build/distributions/**/*",
-    sharedPublicly: true)
+  // sometimes google storage reports ResumableUploadException: 503 Server Error
+  retryWithSleep(retries: 3, seconds: 5, backoff: true) {
+    googleStorageUploadExt(bucket: bucketUri,
+      credentialsId: "${JOB_GCS_EXT_CREDENTIALS}",
+      pattern: "${beatsFolder}/build/distributions/**/*",
+      sharedPublicly: true)
+  }
 }
 
 /**
@@ -704,7 +701,7 @@ def tearDown() {
 */
 def fixPermissions(location) {
   if(isUnix()) {
-    catchError(message: 'There were some failures when fixing the permissions', buildResult: 'SUCCESS', stageResult: 'SUCCESS') {
+    try {
       timeout(5) {
         sh(label: 'Fix permissions', script: """#!/usr/bin/env bash
           set +x
@@ -713,6 +710,8 @@ def fixPermissions(location) {
           docker_setup
           script/fix_permissions.sh ${location}""", returnStatus: true)
       }
+    } catch (Throwable e) {
+      echo "There were some failures when fixing the permissions. ${e.toString()}"
     }
   }
 }

Makefile

@@ -102,6 +102,16 @@ check:
 	@$(MAKE) check-go
 	@$(MAKE) check-no-changes
 
+## check : Run some checks similar to what the default check validation runs in the CI.
+.PHONY: check-default
+check-default:
+	@$(MAKE) check-python
+	@echo "The update goal is skipped to speed up the checks in the CI on a PR basis."
+	@$(MAKE) notice
+	@$(MAKE) check-headers
+	@$(MAKE) check-go
+	@$(MAKE) check-no-changes
+
 ## ccheck-go : Check there is no changes in Go modules.
 .PHONY: check-go
 check-go:

NOTICE.txt

@@ -6345,11 +6345,11 @@ Contents of probable licence file $GOMODCACHE/github.com/elastic/[email protected]/LIC
 
 --------------------------------------------------------------------------------
 Dependency : github.com/elastic/elastic-agent-client/v7
-Version: v7.0.0-20210308165121-7dd05ee2b5a5
+Version: v7.0.0-20210727140539-f0905d9377f6
 Licence type (autodetected): Elastic
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-client/[email protected]20210308165121-7dd05ee2b5a5/LICENSE.txt:
+Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-client/[email protected]20210727140539-f0905d9377f6/LICENSE.txt:
 
 ELASTIC LICENSE AGREEMENT
 
@@ -12643,12 +12643,12 @@ SOFTWARE.
 
 
 --------------------------------------------------------------------------------
-Dependency : github.com/blakerouse/service
-Version: v1.1.1-0.20200924160513-057808572ffa
+Dependency : github.com/kardianos/service
+Version: v1.2.1-0.20210728001519-a323c3813bc7
 Licence type (autodetected): Zlib
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/github.com/blakerouse/service@v1.1.1-0.20200924160513-057808572ffa/LICENSE:
+Contents of probable licence file $GOMODCACHE/github.com/kardianos/service@v1.2.1-0.20210728001519-a323c3813bc7/LICENSE:
 
 Copyright (c) 2015 Daniel Theophanes
 
@@ -12952,11 +12952,11 @@ SOFTWARE.
 
 --------------------------------------------------------------------------------
 Dependency : github.com/miekg/dns
-Version: v1.1.15
+Version: v1.1.25
 Licence type (autodetected): BSD
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/github.com/miekg/[email protected].15/COPYRIGHT:
+Contents of probable licence file $GOMODCACHE/github.com/miekg/[email protected].25/COPYRIGHT:
 
 Copyright 2009 The Go Authors. All rights reserved. Use of this source code
 is governed by a BSD-style license that can be found in the LICENSE file.
@@ -15756,11 +15756,11 @@ Contents of probable licence file $GOMODCACHE/github.com/xdg/[email protected]
 
 --------------------------------------------------------------------------------
 Dependency : go.elastic.co/apm
-Version: v1.8.1-0.20200909061013-2aef45b9cf4b
+Version: v1.11.0
 Licence type (autodetected): Apache-2.0
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/go.elastic.co/apm@v1.8.1-0.20200909061013-2aef45b9cf4b/LICENSE:
+Contents of probable licence file $GOMODCACHE/go.elastic.co/apm@v1.11.0/LICENSE:
 
                                  Apache License
                            Version 2.0, January 2004
@@ -17114,11 +17114,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 --------------------------------------------------------------------------------
 Dependency : golang.org/x/sys
-Version: v0.0.0-20210615035016-665e8c7367d1
+Version: v0.0.0-20210630005230-0f9fa26af87c
 Licence type (autodetected): BSD-3-Clause
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/golang.org/x/[email protected]20210615035016-665e8c7367d1/LICENSE:
+Contents of probable licence file $GOMODCACHE/golang.org/x/[email protected]20210630005230-0f9fa26af87c/LICENSE:
 
 Copyright (c) 2009 The Go Authors. All rights reserved.
 

auditbeat/Jenkinsfile.yml

@@ -35,8 +35,8 @@ stages:
             branches: true     ## for all the branches
             tags: true         ## for all the tags
         stage: extended
-    build:
-        mage: "mage build test"
+    unitTest:
+        mage: "mage build unitTest"
         stage: mandatory
     crosscompile:
         make: "make -C auditbeat crosscompile"

auditbeat/docs/fields.asciidoc

@@ -12676,6 +12676,39 @@ alias to: process.executable
 
 --
 
+[float]
+=== owner
+
+Process owner information.
+
+
+*`process.owner.id`*::
++
+--
+Unique identifier of the user.
+
+type: keyword
+
+--
+
+*`process.owner.name`*::
++
+--
+Short name or login of the user.
+
+type: keyword
+
+example: albert
+
+--
+
+*`process.owner.name.text`*::
++
+--
+type: text
+
+--
+
 [[exported-fields-system]]
 == System fields