Merge remote-tracking branch 'upstream/master' into feature/customise…

…-github-pr-comment-template * upstream/master: [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) Add svc to agent k8s clusterRole (elastic#25146) Add awsfargate module to collect container logs from Amazon ECS on Fargate (elastic#25041) [Filebeat][Cisco ASA] log enhancement and performance (elastic#24744) Watch kubernetes namespaces for autodiscover metadata for pods (elastic#25117) Cyberark Privileged Access Security module (elastic#24803) [Elastic Agent] Log the container command output with LOGS_PATH (elastic#25150) Fix for tests after `device...` field has been removed (elastic#25141) [Ingest Manager] Restart process on output change (elastic#24907) Set --insecure in container when FLEET_SERVER_ENABLE and FLEET_INSECURE set. (elastic#25137) [filebeat] Update documentation / changelog / beta warnings for the syslog input (elastic#25047) Add support for ignore_inactive in filestream input (elastic#25036) Fix bug with annotations dedot config on k8s not used (elastic#25111)
v1v · Apr 20, 2021 · 2d9fb4d · 2d9fb4d
2 parents 6bcaf13 + 2fb65dc
commit 2d9fb4d
Show file tree

Hide file tree

Showing 458 changed files with 34,505 additions and 6,845 deletions.
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
@@ -119,6 +119,7 @@ https://github.com/elastic/beats/compare/v7.11.2...v7.12.0[View commits]
 - Upgrade okta to ECS 1.8.0 and move js processor to ingest pipeline {issue}23118[23118] {pull}23929[23929]
 - Update zoom module to ECS 1.8. {pull}23904[23904] {issue}23118[23118]
 - Add fileset to ingest PostgreSQL CSV logs. {pull}23334[23334]
+- Add beta support for RFC 5424 to the Syslog input. {pull}23954[23954]
 
 *Heartbeat*
 

diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -243,8 +243,10 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Allow cgroup self-monitoring to see alternate `hostfs` paths {pull}24334[24334]
 - Add `expand_keys` to the list of permitted config fields for `decode_json_fields` {24862}[24862]
 - Fix 'make setup' instructions for a new beat {pull}24944[24944]
+- Fix discovery of short-living and failing pods in Kubernetes autodiscover {issue}22718[22718] {pull}24742[24742]
 - Fix inode removal tracking code when files are replaced by files with the same name {pull}25002[25002]
 - Fix `mage GenerateCustomBeat` instructions for a new beat {pull}17679[17679]
+- Fix bug with annotations dedot config on k8s not used {pull}25111[25111]
 - Fix negative Kafka partition bug {pull}25048[25048]
 
 *Auditbeat*
@@ -391,6 +393,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix date parsing in GSuite/login and Google Workspace/login filesets. {issue}24694[24694]
 - Fix date parsing in GSuite/login fileset. {issue}24694[24694]
 - Improve Cisco ASA/FTD parsing of messages - better support for identity FW messages. Change network.bytes, source.bytes, and destination.bytes to long from integer since value can exceed integer capacity.  Add descriptions for various processors for easier pipeline editing in Kibana UI. {pull}23766[23766]
+- Fix usage of unallowed ECS event.outcome values in Cisco ASA/FTD pipeline. {pull}24744[24744].
 - Updating Oauth2 flow for m365_defender fileset. {pull}24829[24829]
 - Improve PanOS parsing and ingest pipeline. {issue}22413[22413] {issue}22748[22748] {pull}24799[24799]
 - Fix S3 input validation for non amazonaws.com domains. {issue}24420[24420] {pull}24861[24861]
@@ -610,10 +613,11 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Added "add_network_direction" processor for determining perimeter-based network direction. {pull}23076[23076]
 - Added new `rate_limit` processor for enforcing rate limits on event throughput. {pull}22883[22883]
 - Allow node/namespace metadata to be disabled on kubernetes metagen and ensure add_kubernetes_metadata honors host {pull}23012[23012]
-- Add `wineventlog` schema to `decode_xml` processor. {issue}23910[23910] {pull}24726[24726]
 - Add new ECS 1.9 field `cloud.service.name` to `add_cloud_metadata` processor. {pull}24993[24993]
 - Libbeat: report queue capacity, output batch size, and output client count to monitoring. {pull}24700[24700]
 - Add kubernetes.pod.ip field in kubernetes metadata. {pull}25037[25037]
+- Discover changes in Kubernetes namespace metadata as soon as they happen. {pull}25117[25117]
+- Add `decode_xml_wineventlog` processor. {issue}23910[23910] {pull}25115[25115]
 
 *Auditbeat*
 
@@ -829,12 +833,15 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Support X-Forwarder-For in IIS logs. {pull}19142[192142]
 - Add support for logs generated by servers configured with `log_statement` and `log_duration` in PostgreSQL module. {pull}24607[24607]
 - Updating field mappings for Cisco AMP module, fixing certain fields. {pull}24661[24661]
+- Added fifteen new message IDs to Cisco ASA/FTD pipeline. {pull}24744[24744]
 - Added NTP fileset to Zeek module {pull}24224[24224]
 - Add `proxy_url` config for httpjson v2 input. {issue}24615[24615] {pull}24662[24662]
 - Add support for upper case field names in Sophos XG module {pull}24693[24693]
 - Add `fail_on_template_error` option for httpjson input. {pull}24784[24784]
 - Change `okta.target` to `flattened` field type.  {issue}24354[24354] {pull}24636[24636]
 - Added `http.request.id` to `nginx/ingress_controller` and `elasticsearch/audit`. {pull}24994[24994]
+- Add `awsfargate` module to collect container logs from Amazon ECS on Fargate. {pull}25041[25041]
+- New module `cyberarkpas` for CyberArk Privileged Access Security audit logs. {pull}24803[24803]
 
 *Heartbeat*
 

diff --git a/auditbeat/docs/fields.asciidoc b/auditbeat/docs/fields.asciidoc
@@ -2477,15 +2477,6 @@ type: keyword
 
 --
 
-*`user_agent.device.type`*::
-+
---
-Type of device where the user agent is running.
-
-type: keyword
-
---
-
 [[exported-fields-cloud]]
 == Cloud provider metadata fields
 

diff --git a/auditbeat/include/fields.go b/auditbeat/include/fields.go
diff --git a/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml b/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml
@@ -561,6 +561,7 @@ rules:
       - namespaces
       - events
       - pods
+      - services
     verbs: ["get", "list", "watch"]
   # Enable this rule only if planing to use kubernetes_secrets provider
   #- apiGroups: [""]

diff --git a/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-role.yaml b/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-role.yaml
@@ -11,6 +11,7 @@ rules:
       - namespaces
       - events
       - pods
+      - services
     verbs: ["get", "list", "watch"]
   # Enable this rule only if planing to use kubernetes_secrets provider
   #- apiGroups: [""]

diff --git a/filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl b/filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl
@@ -320,6 +320,11 @@ filebeat.inputs:
   # Time strings like 2h (2 hours), 5m (5 minutes) can be used.
   #ignore_older: 0
 
+  # Ignore files that have not been updated since the selected event.
+  # ignore_inactive is disabled by default, so no files are ignored by setting it to "".
+  # Available options: since_first_start, since_last_start.
+  #ignore_inactive: ""
+
   # Defines the buffer size every harvester uses when fetching the file
   #harvester_buffer_size: 16384
 
@@ -467,20 +472,22 @@ filebeat.inputs:
   #ssl.client_authentication: "required"
 
 #------------------------------ Syslog input --------------------------------
-# Experimental: Config options for the Syslog input
 # Accept RFC3164 formatted syslog event via UDP.
 #- type: syslog
   #enabled: false
+  #format: rfc3164
   #protocol.udp:
     # The host and port to receive the new event
     #host: "localhost:9000"
 
     # Maximum size of the message received over UDP
     #max_message_size: 10KiB
 
-# Accept RFC3164 formatted syslog event via TCP.
+# Accept RFC5424 formatted syslog event via TCP.
+# RFC5424 support is in beta.
 #- type: syslog
   #enabled: false
+  #format: rfc5424
 
   #protocol.tcp:
     # The host and port to receive the new event

diff --git a/filebeat/autodiscover/builder/hints/logs.go b/filebeat/autodiscover/builder/hints/logs.go
@@ -93,11 +93,6 @@ func (l *logHints) CreateConfig(event bus.Event, options ...ucfg.Option) []*comm
 		return []*common.Config{}
 	}
 
-	host, _ := event["host"].(string)
-	if host == "" {
-		return []*common.Config{}
-	}
-
 	if inputConfig != nil {
 		configs := []*common.Config{}
 		for _, cfg := range inputConfig {