Merge branch 'master' into feature/agent/monitoring-agent-id

CHANGELOG.next.asciidoc

@@ -104,6 +104,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975]
 - Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041]
 - Change logging in logs input to structure logging. Some log message formats have changed. {pull}25299[25299]
+- Change source field for `event.action` in `fortinet.firewall` module to `fortinet.firewall.action` instead of `fortinet.firewall.eventtype`. {pull}24816[24816]
 
 *Heartbeat*
 - Add support for screenshot blocks and use newer synthetics flags that only works in newer synthetics betas. {pull}25808[25808]
@@ -388,6 +389,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add improvements to the azure activitylogs and platformlogs ingest pipelines. {pull}26148[26148]
 - Fix `kibana.log` pipeline when `event.duration` calculation becomes a Long. {issue}24556[24556] {pull}25675[25675]
 - Removed incorrect `http.request.referrer` field from `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
+- Fix `threatintel.indicator.url.full` not being populated. {issue}26351[26351] {pull}26508[26508]
 
 *Heartbeat*
 
@@ -596,6 +598,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Improve ES output error insights. {pull}25825[25825]
 - Add orchestrator.cluster.name/url fields as k8s metadata {pull}26056[26056]
 - Libbeat: report beat version to monitoring. {pull}26214[26214]
+- Ensure common proxy settings support in HTTP clients: proxy_disabled, proxy_url, proxy_headers and typical environment variables HTTP_PROXY, HTTPS_PROXY, NOPROXY. {pull}25219[25219]
 
 *Auditbeat*
 
@@ -811,8 +814,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Update PanOS module to parse Global Protect & User ID logs. {issue}24722[24722] {issue}24724[24724] {pull}24927[24927]
 - Add HMAC signature validation support for http_endpoint input. {pull}24918[24918]
 - Add new grok pattern for iptables module for Ubiquiti UDM {issue}25615[25615] {pull}25616[25616]
-- Add multiline support to aws-s3 input. {issue}25249[25249] {pull}25710[25710]
+- Add multiline support to aws-s3 input. {issue}25249[25249] {pull}25710[25710] {pull}25873[25873]
 - Add monitoring metrics to the `aws-s3` input. {pull}25711[25711]
+- Added `network.direction` fields to Zeek and Suricata modules using the `add_network_direction` processor {pull}24620[24620]
 - Add Content-Type override to aws-s3 input. {issue}25697[25697] {pull}25772[25772]
 - In Cisco Umbrella fileset add users from cisco.umbrella.identities to related.user. {pull}25776[25776]
 - Add fingerprint processor to generate fixed ids for `google_workspace` events. {pull}25841[25841]
@@ -837,10 +841,12 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
 - Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
 - Added dataset `recordedfuture` to the `threatintel` module to ingest indicators from Recorded Future Connect API {pull}26481[26481]
+- Update `fortinet` ingest pipelines. {issue}22136[22136] {issue}25254[25254] {pull}24816[24816]
 
 *Heartbeat*
 
 - Add mime type detection for http responses. {pull}22976[22976]
+- Add `proxy_headers` to HTTP monitor. {pull}25219[25219]
 
 *Journalbeat*
 

NOTICE.txt

@@ -16717,11 +16717,11 @@ THE SOFTWARE.
 
 --------------------------------------------------------------------------------
 Dependency : golang.org/x/crypto
-Version: v0.0.0-20200820211705-5c72a883971a
+Version: v0.0.0-20210616213533-5ff15b29337e
 Licence type (autodetected): BSD-3-Clause
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/golang.org/x/[email protected]20200820211705-5c72a883971a/LICENSE:
+Contents of probable licence file $GOMODCACHE/golang.org/x/[email protected]20210616213533-5ff15b29337e/LICENSE:
 
 Copyright (c) 2009 The Go Authors. All rights reserved.
 
@@ -16902,11 +16902,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 --------------------------------------------------------------------------------
 Dependency : golang.org/x/sys
-Version: v0.0.0-20210308170721-88b6017d0656
+Version: v0.0.0-20210615035016-665e8c7367d1
 Licence type (autodetected): BSD-3-Clause
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/golang.org/x/[email protected]20210308170721-88b6017d0656/LICENSE:
+Contents of probable licence file $GOMODCACHE/golang.org/x/[email protected]20210615035016-665e8c7367d1/LICENSE:
 
 Copyright (c) 2009 The Go Authors. All rights reserved.
 

dev-tools/cmd/dashboards/export_dashboards.go

@@ -29,6 +29,7 @@ import (
 
 	"github.com/pkg/errors"
 
+	"github.com/elastic/beats/v7/libbeat/common/transport/httpcommon"
 	"github.com/elastic/beats/v7/libbeat/dashboards"
 	"github.com/elastic/beats/v7/libbeat/kibana"
 )
@@ -64,14 +65,18 @@ func main() {
 		user = u.User.Username()
 		pass, _ = u.User.Password()
 	}
+
+	transport := httpcommon.DefaultHTTPTransportSettings()
+	transport.Timeout = kibanaTimeout
+
 	client, err := kibana.NewClientWithConfig(&kibana.ClientConfig{
-		Protocol: u.Scheme,
-		Host:     u.Host,
-		Username: user,
-		Password: pass,
-		Path:     u.Path,
-		SpaceID:  *spaceID,
-		Timeout:  kibanaTimeout,
+		Protocol:  u.Scheme,
+		Host:      u.Host,
+		Username:  user,
+		Password:  pass,
+		Path:      u.Path,
+		SpaceID:   *spaceID,
+		Transport: transport,
 	})
 	if err != nil {
 		log.Fatalf("Error while connecting to Kibana: %v", err)

filebeat/docs/faq.asciidoc

@@ -5,6 +5,13 @@ This section describes common problems you might encounter with
 {beatname_uc}. Also check out the
 https://discuss.elastic.co/c/beats/{beatname_lc}[{beatname_uc} discussion forum].
 
+[[filebeat-kubernetes-metadata-error-extracting-container-id]]
+=== Error extracting container id while using Kubernetes metadata
+
+The `add_kubernetes_metadata` processor might throw the error `Error extracting container id - source value does not contain matcher's logs_path`.
+There might be some issues with the matchers definitions or the location of `logs_path`.
+Please verify the Kubernetes pod is healthy.
+
 [[filebeat-network-volumes]]
 === Can't read log files from network volumes
 

filebeat/docs/fields.asciidoc

@@ -62883,6 +62883,16 @@ type: keyword
 ESP Transform
 
 
+type: keyword
+
+--
+
+*`fortinet.firewall.eventtype`*::
++
+--
+UTM Event Type
+
+
 type: keyword
 
 --
@@ -65363,6 +65373,16 @@ type: integer
 Security action performed by UTM
 
 
+type: keyword
+
+--
+
+*`fortinet.firewall.utmref`*::
++
+--
+Reference to UTM
+
+
 type: keyword
 
 --

filebeat/docs/modules/suricata.asciidoc

@@ -51,6 +51,15 @@ A list of tags to include in events. Including `forwarded` indicates that the
 events did not originate on this host and causes `host.name` to not be added to
 events. Defaults to `[suricata]`.
 
+`var.internal_networks`::
+
+A list of CIDR ranges describing the IP addresses that
+you consider internal. This is used in determining the value of
+`network.direction`. The values
+can be either a CIDR value or one of the named ranges supported by the
+<<condition-network, `network`>> condition. The default value is `[private]`
+which classifies RFC 1918 (IPv4) and RFC 4193 (IPv6) addresses as internal.
+
 [float]
 === Example dashboard