From b332fc28794083e360eac220a5150b443dec6d15 Mon Sep 17 00:00:00 2001 From: Elastic Machine Date: Tue, 25 May 2021 06:05:30 -0400 Subject: [PATCH] docs: Prepare Changelog for 7.13.0 (#25823) * docs: Close changelog for 7.13.0 * Cleanup * Cleanup * Move 23201 to the correct place * Apply suggestions from code review Co-authored-by: Jaime Soriano Pastor * Move 24123 * Remove empty sections * Add missing 7.12.1 * Additional fixes Co-authored-by: Andres Rodriguez Co-authored-by: Andres Rodriguez Co-authored-by: Jaime Soriano Pastor --- CHANGELOG.asciidoc | 177 +++++++++++++++++++++++++ CHANGELOG.next.asciidoc | 237 ---------------------------------- libbeat/docs/release.asciidoc | 1 + 3 files changed, 178 insertions(+), 237 deletions(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 269a69546ca7..801d8641b3af 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -3,6 +3,183 @@ :issue: https://github.com/elastic/beats/issues/ :pull: https://github.com/elastic/beats/pull/ +[[release-notes-7.13.0]] +=== Beats version 7.13.0 +https://github.com/elastic/beats/compare/v7.12.1...v7.13.0[View commits] + +==== Breaking changes + +*Affecting all Beats* + +- Use alias to report container image in k8s metadata. {pull}24380[24380] +- Set `cleanup_timeout` to zero by default in docker and kubernetes autodiscover in all beats except Filebeat where it is kept to 60 seconds. {pull}24681[24681] +- Update to ECS 1.9.0. {pull}24909[24909] + +*Filebeat* + +- Changes filebeat httpjson input's append transform to create a list even with only a single value{pull}25074[25074] +- Deprecated the cyberark module (replaced by cyberarkpas). {issue}25261[25261] {pull}25505[25505] + +*Metricbeat* + +- Store `cloudfoundry.container.cpu.pct` in decimal form and as `scaled_float`. {pull}24219[24219] +- Remove `index_stats.created` field from Elasticsearch/index Metricset {pull}25113[25113] + +==== Bugfixes + +*Affecting all Beats* + +- Fix events being dropped if they contain a floating point value of NaN or Inf. {pull}25051[25051] +- Fix templates being overwritten if there was an error when check for the template existance. {pull}24332[24332] +- Add `expand_keys` to the list of permitted config fields for `decode_json_fields` {24862}[24862] +- Fix discovery of short-living and failing pods in Kubernetes autodiscover {issue}22718[22718] {pull}24742[24742] +- Fix panic when overwriting metadata {pull}24741[24741] +- Fix role_arn to work with access keys for AWS. {pull}25446[25446] +- Fix `community_id` processor so that ports greater than 65535 aren't valid. {pull}25409[25409] + +*Auditbeat* + +- Fix o365 module config when client_secret contains special characters. {issue}25058[25058] + +*Filebeat* + +- Fix date parsing in GSuite/login fileset. {issue}24694[24694] +- Improve Cisco ASA/FTD parsing of messages {pull}23766[23766] + - Better support for identity FW messages. + - Change network.bytes, source.bytes, and destination.bytes to long from integer since value can exceed integer capacity. + - Add descriptions for various processors for easier pipeline editing in Kibana UI. +- Fix usage of unallowed ECS event.outcome values in Cisco ASA/FTD pipeline. {pull}24744[24744]. +- Fix IPtables Pipeline and Ubiquiti dashboard. {issue}24878[24878] {pull}24928[24928] +- Strip Azure Eventhub connection string in debug logs. {pulll}25066[25066] +- Updating Oauth2 flow for m365_defender fileset. {pull}24829[24829] +- Fix o365 module config when client_secret contains special characters. {issue}25058[25058] +- Fix s3 input when there is a blank line in the log file. {pull}25357[25357] +- Remove space from field `sophos.xg.trans_src_ ip`. {issue}25154[25154] {pull}25250[25250] +- Fix `checkpoint.action_reason` when its a string, not a Long. {issue}25575[25575] {pull}25609[25609] +- Fix `fortinet.firewall.addr` when its a string, not an IP address. {issue}25585[25585] {pull}25608[25608] + +*Metricbeat* + +- Sort correctly the keys when accessing JMX through the Jolokia module {pull}25631[25631] +- Change lookup_fields from metricset.host to service.address {pull}15883[15883] +- Fix incorrect types of fields GetHits and Ops in NodeInterestingStats for Couchbase module in Metricbeat {issue}21021[21021] {pull}23287[23287] +- Fix GCP not able to request Cloudfunctions metrics if a region filter was set {pull}24218[24218] +- Fix type of `uwsgi.status.worker.rss` type. {pull}24468[24468] +- Accept text/plain type by default for prometheus client scraping. {pull}24622[24622] +- Use working set bytes to calculate the pod memory limit pct when memory usage is not reported (ie. Windows pods). {pull}25428[25428] +- Fix copy-paste error in libbeat docs. {pull}25448[25448] +- Fix azure billing dashboard. {pull}25554[25554] + +*Winlogbeat* + +- Change `event.code` and `winlog.event_id` from int to keyword. {pull}25176[25176] + +==== Added + +*Affecting all Beats* + +- Add `wineventlog` schema to `decode_xml` processor. {issue}23910[23910] {pull}24726[24726] +- Add new ECS 1.9 field `cloud.service.name` to `add_cloud_metadata` processor. {pull}24993[24993] +- Libbeat: report queue capacity, output batch size, and output client count to monitoring. {pull}24700[24700] +- Add kubernetes.pod.ip field in kubernetes metadata. {pull}25037[25037] +- Discover changes in Kubernetes namespace metadata as soon as they happen. {pull}25117[25117] +- Add `decode_xml_wineventlog` processor. {issue}23910[23910] {pull}25115[25115] +- Add new setting `gc_percent` for tuning the garbage collector limits via configuration file. {pull}25394[25394] +- Add `unit` and `metric_type` properties to fields.yml for populating field metadata in Elasticsearch templates {pull}25419[25419] +- Add new option `suffix` to `logging.files` to control how log files are rotated. {pull}25464[25464] +- Validate that required functionality in Elasticsearch is available upon initial connection. {pull}25351[25351] + +*Filebeat* + +- Support X-Forwarder-For in IIS logs. {pull}19142[192142] +- Add support for logs generated by servers configured with `log_statement` and `log_duration` in PostgreSQL module. {pull}24607[24607] +- Added fifteen new message IDs to Cisco ASA/FTD pipeline. {pull}24744[24744] +- Added NTP fileset to Zeek module {pull}24224[24224] +- Add `proxy_url` config for httpjson v2 input. {issue}24615[24615] {pull}24662[24662] +- Change `okta.target` to `flattened` field type. {issue}24354[24354] {pull}24636[24636] +- Added `http.request.id` to `nginx/ingress_controller` and `elasticsearch/audit`. {pull}24994[24994] +- Add `awsfargate` module to collect container logs from Amazon ECS on Fargate. {pull}25041[25041] +- New module `cyberarkpas` for CyberArk Privileged Access Security audit logs. {pull}24803[24803] +- Add `uri_parts` processor to Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules ingest pipelines. {issue}19088[19088] {pull}24699[24699] +- New module `zookeeper` for Zookeeper service and audit logs {issue}25061[25061] {pull}25128[25128] +- Add parsing for `haproxy.http.request.raw_request_line` field {issue}25480[25480] {pull}25482[25482] +- Mark `filestream` input beta. {pull}25560[25560] +- Add User Agent Parser for Azure Sign In Logs Ingest Pipeline {pull}23201[23201] + +*Heartbeat* + +- Handle datastreams for fleet. {pull}24223[24223] +- Add --sandbox option for browser monitor. {pull}24172[24172] +- Support additional 'root' fields from synthetics. {pull}24770[24770] +- Browser zip_url source type. {pull}24714[24714] + +*Metricbeat* + +- Add support for Consul 1.9. {pull}24123[24123] +- Add support for defining metrics_filters for prometheus module in hints. {pull}24264[24264] +- Add support for PostgreSQL 10, 11, 12 and 13. {pull}24402[24402] +- Add support for SASL/SCRAM authentication to the Kafka module. {pull}24810[24810] + +*Winlogbeat* + +- Add support for sysmon v13 events 24 and 25. {issue}24217[24217] {pull}24945[24945] + + +[[release-notes-7.12.1]] +=== Beats version 7.12.1 +https://github.com/elastic/beats/compare/v7.12.0...v7.12.1[View commits] + +==== Breaking changes + +*Filebeat* + +- Possible values for Netflow's locality fields (source.locality, destination.locality and flow.locality) are now `internal` and `external`, instead of `private` and `public`. {issue}24272[24272] {pull}24295[24295] + +==== Bugfixes + +*Affecting all Beats* + +- Fix templates being overwritten if there was an error when check for the template existance. {pull}24332[24332] +- Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data {pull}17223[17223] +- Fix inode removal tracking code when files are replaced by files with the same name {pull}25002[25002] +- Fix `mage GenerateCustomBeat` instructions for a new beat {pull}17679[17679] +- Fix bug with annotations dedot config on k8s not used {pull}25111[25111] +- Fix negative Kafka partition bug {pull}25048[25048] + +*Filebeat* + +- Properly update offset in case of unparasable line. {pull}22685[22685] +- Fix Cisco ASA parser for message 722051. {pull}24410[24410] +- Fix `google_workspace` pagination. {pull}24668[24668] +- Fix netflow module ignoring detect_sequence_reset flag. {issue}24268[24268] {pull}24270[24270] +- Fix Cisco ASA parser for message 302022. {issue}24405[24405] {pull}24697[24697] +- Fix Cisco AMP `@metadata._id` calculation {issue}24717[24717] {pull}24718[24718] +- Fix date parsing in GSuite/login and Google Workspace/login filesets. {issue}24694[24694] +- Fix gcp/vpcflow module error where input type was defaulting to file. {pull}24719[24719] +- Improve PanOS parsing and ingest pipeline. {issue}22413[22413] {issue}22748[22748] {pull}24799[24799] +- Fix S3 input validation for non amazonaws.com domains. {issue}24420[24420] {pull}24861[24861] +- Fix google_workspace and okta modules pagination when next page template is empty. {pull}24967[24967] +- Fix gcp module field names to use gcp instead of googlecloud. {pull}25038[25038] + +*Heartbeat* + +- Fix panic when initialization of ICMP monitors fail twice. {pull}25073[25073] + +*Metricbeat* + +- Ignore unsupported derive types for filesystem metricset. {issue}22501[22501] {pull}24502[24502] + + +==== Added + +*Filebeat* + +- Updating field mappings for Cisco AMP module, fixing certain fields. {pull}24661[24661] +- Add support for upper case field names in Sophos XG module {pull}24693[24693] +- Add `fail_on_template_error` option for httpjson input. {pull}24784[24784] + + + [[release-notes-7.12.0]] === Beats version 7.12.0 https://github.com/elastic/beats/compare/v7.11.2...v7.12.0[View commits] diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index f2d091d3baaa..1cd7459a53e0 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -19,9 +19,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Introduce APM libbeat instrumentation, active when running the beat with ELASTIC_APM_ACTIVE=true. {pull}17938[17938] - Make error message about locked data path actionable. {pull}18667[18667] - Fix panic with inline SSL when the certificate or key were small than 256 bytes. {pull}23820[23820] -- Use alias to report container image in k8s metadata. {pull}24380[24380] -- Set `cleanup_timeout` to zero by default in docker and kubernetes autodiscover in all beats except Filebeat where it is kept to 60 seconds. {pull}24681[24681] -- Update to ECS 1.9.0. {pull}24909[24909] *Auditbeat* @@ -44,16 +41,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Disable the option of running --machine-learning on its own. {pull}20241[20241] - Fix PANW field spelling "veredict" to "verdict" on event.action {pull}18808[18808] - Add support for GMT timezone offsets in `decode_cef`. {pull}20993[20993] -- API address and shard ID are required settings in the Cloud Foundry input. {pull}21759[21759] -- Remove `suricata.eve.timestamp` alias field. {issue}10535[10535] {pull}22095[22095] -- Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. {pull}22571[22571] -- Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975] -- Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041] -- Rename `s3` input to `aws-s3` input. {pull}23469[23469] -- Possible values for Netflow's locality fields (source.locality, destination.locality and flow.locality) are now `internal` and `external`, instead of `private` and `public`. {issue}24272[24272] {pull}24295[24295] -- Add User Agent Parser for Azure Sign In Logs Ingest Pipeline {pull}23201[23201] -- Changes filebeat httpjson input's append transform to create a list even with only a single value{pull}25074[25074] -- Deprecated the cyberark module (replaced by cyberarkpas). {issue}25261[25261] {pull}25505[25505] *Heartbeat* @@ -68,16 +55,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. {issue}11975[11975] - Fix ECS compliance of user.id field in system/users metricset {pull}19019[19019] - Remove "invalid zero" metrics on Windows and Darwin, don't report linux-only memory and diskio metrics when running under agent. {pull}21457[21457] -- Change cloud.provider from googlecloud to gcp. {pull}21775[21775] -- API address and shard ID are required settings in the Cloud Foundry module. {pull}21759[21759] -- Rename googlecloud module to gcp module. {pull}22246[22246] -- Use ingress/egress instead of inbound/outbound for system/socket metricset. {pull}22992[22992] -- Change types of numeric metrics from Kubelet summary api to double so as to cover big numbers. {pull}23335[23335] -- Add container.image.name and containe.name ECS fields for state_container. {pull}23802[23802] -- Add support for Consul 1.9. {pull}24123[24123] -- Add support for the MemoryPressure, DiskPressure, OutOfDisk and PIDPressure status conditions in state_node. {pull}23905[23905] -- Store `cloudfoundry.container.cpu.pct` in decimal form and as `scaled_float`. {pull}24219[24219] -- Remove `index_stats.created` field from Elasticsearch/index Metricset {pull}25113[25113] *Packetbeat* @@ -102,8 +79,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Affecting all Beats* -- Fix events being dropped if they contain a floating point value of NaN or Inf. {pull}25051[25051] -- Fix templates being overwritten if there was an error when check for the template existance. {pull}24332[24332] - Fix a race condition with the Kafka pipeline client, it is possible that `Close()` get called before `Connect()` . {issue}11945[11945] - Allow users to configure only `cluster_uuid` setting under `monitoring` namespace. {pull}14338[14338] - Update replicaset group to apps/v1 {pull}15854[15802] @@ -137,28 +112,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - [Metricbeat][Kubernetes] Change cluster_ip field from ip to keyword. {pull}20571[20571] - The `o365input` and `o365` module now recover from an authentication problem or other fatal errors, instead of terminating. {pull}21258[21258] - Periodic metrics in logs will now report `libbeat.output.events.active` and `beat.memstats.rss` - as gauges (rather than counters). {pull}22877[22877] -- Use PROGRAMDATA environment variable instead of C:\ProgramData for windows install service {pull}22874[22874] -- Fix reporting of cgroup metrics when running under Docker {pull}22879[22879] -- Fix typo in config docs {pull}23185[23185] -- Fix `nested` subfield handling in generated Elasticsearch templates. {issue}23178[23178] {pull}23183[23183] -- Fix CPU usage metrics on VMs with dynamic CPU config {pull}23154[23154] -- Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete {pull}23419[23419] -- Fix error loop with runaway CPU use when the Kafka output encounters some connection errors {pull}23484[23484] -- Allow configuring credential_profile_name and shared_credential_file when using role_arn. {pull}24174[24174] -- Fix ILM setup log reporting that a policy or an alias was created, even though the creation of any resource was disabled. {issue}24046[24046] {pull}24480[24480] -- Fix ILM alias not being created if `setup.ilm.check_exists: false` and `setup.ilm.overwrite: true` has been configured. {pull}24480[24480] -- Allow cgroup self-monitoring to see alternate `hostfs` paths {pull}24334[24334] - -- Add `expand_keys` to the list of permitted config fields for `decode_json_fields` {24862}[24862] -- Fix 'make setup' instructions for a new beat {pull}24944[24944] -- Fix discovery of short-living and failing pods in Kubernetes autodiscover {issue}22718[22718] {pull}24742[24742] -- Fix inode removal tracking code when files are replaced by files with the same name {pull}25002[25002] -- Fix negative Kafka partition bug {pull}25048[25048] -- Fix bug with annotations dedot config on k8s not used {pull}25111[25111] -- Fix panic when overwriting metadata {pull}24741[24741] -- Fix role_arn to work with access keys for AWS. {pull}25446[25446] -- Fix `community_id` processor so that ports greater than 65535 aren't valid. {pull}25409[25409] *Auditbeat* @@ -217,80 +170,23 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Fix event.kind for system/syslog pipeline {issue}20365[20365] {pull}20390[20390] - Fix event.type for zeek/ssl and duplicate event.category for zeek/connection {pull}20696[20696] - Add json body check for sqs message. {pull}21727[21727] -- Properly update offset in case of unparasable line. {pull}22685[22685] - Drop aws.vpcflow.pkt_srcaddr and aws.vpcflow.pkt_dstaddr when equal to "-". {pull}22721[22721] {issue}22716[22716] -- Fix cisco umbrella module config by adding input variable. {pull}22892[22892] -- Fix network.direction logic in zeek connection fileset. {pull}22967[22967] -- Fix aws s3 overview dashboard. {pull}23045[23045] -- Fix bad `network.direction` values in Fortinet/firewall fileset. {pull}23072[23072] -- Fix Cisco ASA/FTD module's parsing of WebVPN log message 716002. {pull}22966[22966] -- Add support for organization and custom prefix in AWS/CloudTrail fileset. {issue}23109[23109] {pull}23126[23126] -- Simplify regex for organization custom prefix in AWS/CloudTrail fileset. {issue}23203[23203] {pull}23204[23204] -- Fix syslog header parsing in infoblox module. {issue}23272[23272] {pull}23273[23273] -- Fix CredentialsJSON unpacking for `gcp-pubsub` and `httpjson` inputs. {pull}23277[23277] -- Fix concurrent modification exception in Suricata ingest node pipeline. {pull}23534[23534] -- Change the `event.created` in Netflow events to be the time the event was created by Filebeat - to be consistent with ECS. {pull}23094[23094] -- Fix Zoom module parameters for basic auth and url path. {pull}23779[23779] -- Fix handling of ModifiedProperties field in Office 365. {pull}23777[23777] -- Use rfc6587 framing for fortinet firewall and clientendpoint filesets when transferring over tcp. {pull}23837[23837] -- Fix httpjson input logging so it doesn't conflict with ECS. {pull}23972[23972] -- Fix Logstash module handling of logstash.log.log_event.action field. {issue}20709[20709] -- aws/s3access dataset was populating event.duration using the wrong unit. {pull}23920[23920] -- Zoom module pipeline failed to ingest some chat_channel events. {pull}23904[23904] -- Fix Netlow module issue with missing `internal_networks` config parameter. {issue}24094[24094] {pull}24110[24110] -- in httpjson input using encode_as "application/x-www-form-urlencoded" now sets Content-Type correctly {issue}24331[24331] {pull}24336[24336] -- Fix default `scope` in `add_nomad_metadata`. {issue}24559[24559] -- Fix Cisco ASA parser for message 722051. {pull}24410[24410] -- Fix `google_workspace` pagination. {pull}24668[24668] -- Fix netflow module ignoring detect_sequence_reset flag. {issue}24268[24268] {pull}24270[24270] -- Fix Cisco ASA parser for message 302022. {issue}24405[24405] {pull}24697[24697] -- Fix Cisco AMP `@metadata._id` calculation {issue}24717[24717] {pull}24718[24718] -- Fix date parsing in GSuite/login and Google Workspace/login filesets. {issue}24694[24694] -- Fix gcp/vpcflow module error where input type was defaulting to file. {pull}24719[24719] -- Fix gcp/vpcflow module error where input type was defaulting to file. {pull}24719[24719] -- Fix date parsing in GSuite/login and Google Workspace/login filesets. {issue}24694[24694] -- Fix date parsing in GSuite/login fileset. {issue}24694[24694] -- Improve Cisco ASA/FTD parsing of messages - better support for identity FW messages. Change network.bytes, source.bytes, and destination.bytes to long from integer since value can exceed integer capacity. Add descriptions for various processors for easier pipeline editing in Kibana UI. {pull}23766[23766] -- Fix usage of unallowed ECS event.outcome values in Cisco ASA/FTD pipeline. {pull}24744[24744]. -- Updating Oauth2 flow for m365_defender fileset. {pull}24829[24829] -- Improve PanOS parsing and ingest pipeline. {issue}22413[22413] {issue}22748[22748] {pull}24799[24799] -- Fix S3 input validation for non amazonaws.com domains. {issue}24420[24420] {pull}24861[24861] -- Fix google_workspace and okta modules pagination when next page template is empty. {pull}24967[24967] -- Fix IPtables Pipeline and Ubiquiti dashboard. {issue}24878[24878] {pull}24928[24928] -- Fix gcp module field names to use gcp instead of googlecloud. {pull}25038[25038] -- Strip Azure Eventhub connection string in debug logs. {pulll}25066[25066] -- Updating Oauth2 flow for m365_defender fileset. {pull}24829[24829] -- Fix o365 module config when client_secret contains special characters. {issue}25058[25058] -- Fix s3 input when there is a blank line in the log file. {pull}25357[25357] -- Remove space from field `sophos.xg.trans_src_ ip`. {issue}25154[25154] {pull}25250[25250] -- Fix `checkpoint.action_reason` when its a string, not a Long. {issue}25575[25575] {pull}25609[25609] -- Fix `fortinet.firewall.addr` when its a string, not an IP address. {issue}25585[25585] {pull}25608[25608] *Heartbeat* - Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. {pull}15639[15639] - Fixed scheduler shutdown issues which would in rare situations cause a panic due to semaphore misuse. {pull}16397[16397] - Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. {pull}17549[17549] -- Fix panic when initialization of ICMP monitors fail twice. {pull}25073[25073] *Journalbeat* *Metricbeat* -- Sort correctly the keys when accessing JMX through the Jolokia module {pull}25631[25631] -- Add dedot for tags in ec2 metricset and cloudwatch metricset. {issue}15843[15843] {pull}15844[15844] -- Use RFC3339 format for timestamps collected using the SQL module. {pull}15847[15847] -- Avoid parsing errors returned from prometheus endpoints. {pull}15712[15712] -- Change lookup_fields from metricset.host to service.address {pull}15883[15883] -- Add dedot for cloudwatch metric name. {issue}15916[15916] {pull}15917[15917] -- Fixed issue `logstash-xpack` module suddenly ceasing to monitor Logstash. {issue}15974[15974] {pull}16044[16044] - Fix checking tagsFilter using length in cloudwatch metricset. {pull}14525[14525] - Fixed bug with `elasticsearch/cluster_stats` metricset not recording license expiration date correctly. {issue}14541[14541] {pull}14591[14591] - Log bulk failures from bulk API requests to monitoring cluster. {issue}14303[14303] {pull}14356[14356] - Fixed bug with `elasticsearch/cluster_stats` metricset not recording license ID in the correct field. {pull}14592[14592] -- Change lookup_fields from metricset.host to service.address {pull}15883[15883] - Fix skipping protocol scheme by light modules. {pull}16205[pull] - Made `logstash-xpack` module once again have parity with internally-collected Logstash monitoring data. {pull}16198[16198] - Revert changes in `docker` module: add size flag to docker.container. {pull}16600[16600] @@ -335,20 +231,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Remove io.time from windows {pull}22237[22237] - Fix `logstash` module when `xpack.enabled: true` is set from emitting redundant events. {pull}22808[22808] - Change vsphere.datastore.capacity.used.pct value to betweeen 0 and 1. {pull}23148[23148] -- Fix incorrect types of fields GetHits and Ops in NodeInterestingStats for Couchbase module in Metricbeat {issue}21021[21021] {pull}23287[23287] -- Update config in `windows.yml` file. {issue}23027[23027]{pull}23327[23327] -- Add stack monitoring section to elasticsearch module documentation {pull}#23286[23286] -- Fix metric grouping for windows/perfmon module {issue}23489[23489] {pull}23505[23505] -- Fix ec2 metricset fields.yml and the integration test {pull}23726[23726] -- Unskip s3_request integration test. {pull}23887[23887] -- Add system.hostfs configuration option for system module. {pull}23831[23831] -- Fix GCP not able to request Cloudfunctions metrics if a region filter was set {pull}24218[24218] -- Fix type of `uwsgi.status.worker.rss` type. {pull}24468[24468] -- Ignore unsupported derive types for filesystem metricset. {issue}22501[22501] {pull}24502[24502] -- Accept text/plain type by default for prometheus client scraping. {pull}24622[24622] -- Use working set bytes to calculate the pod memory limit pct when memory usage is not reported (ie. Windows pods). {pull}25428[25428] -- Fix copy-paste error in libbeat docs. {pull}25448[25448] -- Fix azure billing dashboard. {pull}25554[25554] *Packetbeat* @@ -356,7 +238,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Winlogbeat* -- Change `event.code` and `winlog.event_id` from int to keyword. {pull}25176[25176] *Functionbeat* @@ -395,16 +276,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Add capability of enriching process metadata with contianer id also for non-privileged containers in `add_process_metadata` processor. {pull}19767[19767] - Add replace_fields config option in add_host_metadata for replacing host fields. {pull}20490[20490] {issue}20464[20464] - Add option to select the type of index template to load: legacy, component, index. {pull}21212[21212] -- Add `wineventlog` schema to `decode_xml` processor. {issue}23910[23910] {pull}24726[24726] -- Add new ECS 1.9 field `cloud.service.name` to `add_cloud_metadata` processor. {pull}24993[24993] -- Libbeat: report queue capacity, output batch size, and output client count to monitoring. {pull}24700[24700] -- Add kubernetes.pod.ip field in kubernetes metadata. {pull}25037[25037] -- Discover changes in Kubernetes namespace metadata as soon as they happen. {pull}25117[25117] -- Add `decode_xml_wineventlog` processor. {issue}23910[23910] {pull}25115[25115] -- Add new setting `gc_percent` for tuning the garbage collector limits via configuration file. {pull}25394[25394] -- Add `unit` and `metric_type` properties to fields.yml for populating field metadata in Elasticsearch templates {pull}25419[25419] -- Add new option `suffix` to `logging.files` to control how log files are rotated. {pull}25464[25464] -- Validate that required functionality in Elasticsearch is available upon initial connection. {pull}25351[25351] *Auditbeat* @@ -494,101 +365,15 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Keep cursor state between httpjson input restarts {pull}20751[20751] - New juniper.srx dataset for Juniper SRX logs. {pull}20017[20017] - Added DNS response IP addresses to `related.ip` in Suricata module. {pull}22291[22291] -- Added TLS JA3 fingerprint, certificate not_before/not_after, certificate SHA1 hash, and certificate subject fields to Zeek SSL dataset. {pull}21696[21696] -- Add platform logs in the azure filebeat module. {pull}22371[22371] -- Added `event.ingested` field to data from the Netflow module. {pull}22412[22412] -- Improve panw ECS url fields mapping. {pull}22481[22481] -- Improve Nats filebeat dashboard. {pull}22726[22726] -- Add support for UNIX datagram sockets in `unix` input. {issues}18632[18632] {pull}22699[22699] -- Add `http.request.mime_type` for Elasticsearch audit log fileset. {pull}22975[22975] -- Add new httpjson input features and mark old config ones for deprecation {pull}22320[22320] -- Add configuration option to set external and internal networks for panw panos fileset {pull}22998[22998] -- Add `subbdomain` fields for rsa2elk modules. {pull}23035[23035] -- Add subdomain enrichment for suricata/eve fileset. {pull}23011[23011] -- Add subdomain enrichment for zeek/dns fileset. {pull}23011[23011] -- Add `event.category` "configuration" to auditd module events. {pull}23010[23010] -- Add `event.category` "configuration" to gsuite module events. {pull}23010[23010] -- Add `event.category` "configuration" to o365 module events. {pull}23010[23010] -- Add `event.category` "configuration" to zoom module events. {pull}23010[23010] -- Add `network.direction` to auditd/log fileset. {pull}23041[23041] -- Add logic for external network.direction in sophos xg fileset {pull}22973[22973] -- Preserve AWS CloudTrail eventCategory in aws.cloudtrail.event_category. {issue}22776[22776] {pull}22805[22805] -- Add top_level_domain enrichment for suricata/eve fileset. {pull}23046[23046] -- Add top_level_domain enrichment for zeek/dns fileset. {pull}23046[23046] -- Add `observer.egress.zone` and `observer.ingress.zone` for cisco/asa and cisco/ftd filesets. {pull}23068[23068] -- Allow cisco/asa and cisco/ftd filesets to override network directionality based off of zones. {pull}23068[23068] -- Allow cef and checkpoint modules to override network directionality based off of zones {pull}23066[23066] -- Add `network.direction` to netflow/log fileset. {pull}23052[23052] -- Add the ability to override `network.direction` based on interfaces in Fortinet/firewall fileset. {pull}23072[23072] -- Add `network.direction` override by specifying `internal_networks` in gcp module. {pull}23081[23081] -- Migrate microsoft/defender_atp to httpjson v2 config {pull}23017[23017] -- Migrate microsoft/m365_defender to httpjson v2 config {pull}23018[23018] -- Migrate okta to httpjson v2 config {pull}23059[23059] -- Add support for Snyk Vulnerability and Audit API. {pull}22677[22677] -- Misp improvements: Migration to httpjson v2 config, pagination and deduplication ID {pull}23070[23070] -- Add Google Workspace module and mark Gsuite module as deprecated {pull}22950[22950] -- Mark m365 defender, defender atp, okta and google workspace modules as GA {pull}23113[23113] -- Add parsing of tcp flags to AWS vpcflow fileset {issue}228020[22820] {pull}23157[23157] - Added support for first_event context in filebeat httpjson input {pull}23437[23437] -- Added `alternative_host` option to google pubsub input {pull}23215[23215] -- Adding Threat Intel module {pull}21795[21795] -- Added username parsing from Cisco ASA message 302013. {pull}21196[21196] -- Added `encode_as` and `decode_as` options to httpjson along with pluggable encoders/decoders {pull}23478[23478] - Added feature to modules to adapt Ingest Node pipelines for compatibility with older Elasticsearch versions by - removing unsupported processors. {pull}23763[23763] -- Added support for Cisco AMP API as a new fileset. {pull}22768[22768] -- Added RFC6587 framing option for tcp and unix inputs {issue}23663[23663] {pull}23724[23724] -- Added `application/x-ndjson` as decode option for httpjson input {pull}23521[23521] -- Added `application/x-www-form-urlencoded` as encode option for httpjson input {pull}23521[23521] -- Move aws-s3 input to GA. {pull}23631[23631] -- Add support for upper case field names in Sophos XG module {pull}24693[24693] -- Populate `source.mac` and `destination.mac` for Suricata EVE events. {issue}23706[23706] {pull}23721[23721] -- Added string splitting for httpjson input {pull}24022[24022] -- Added Signatures fileset to Zeek module {pull}23772[23772] -- Upgrade Cisco ASA/FTD/Umbrella to ECS 1.8.0. {pull}23819[23819] -- Add new ECS user and categories features to google_workspace/gsuite {issue}23118[23118] {pull}23709[23709] -- Move crowdstrike JS processor to ingest pipelines and upgrade to ECS 1.8.0 {issue}23118[23118] {pull}23875[23875] -- Update Filebeat auditd dataset to ECS 1.8.0. {pull}23723[23723] {issue}23118[23118] -- Updated microsoft defender_atp and m365_defender to ECS 1.8. {pull}23897[23897] {issue}23118[23118] -- Updated o365 module to ECS 1.8. {issue}23118[23118] {pull}23896[23896] -- Upgrade CEF module to ECS 1.8.0. {pull}23832[23832] -- Upgrade fortinet/firewall to ECS 1.8 {issue}23118[23118] {pull}23902[23902] -- Upgrade Zeek to ECS 1.8.0. {issue}23118[23118] {pull}23847[23847] -- Updated azure module to ECS 1.8. {issue}23118[23118] {pull}23927[23927] -- Update aws/s3access to ECS 1.8. {issue}23118[23118] {pull}23920[23920] - Upgrade panw module to ecs 1.8 {issue}23118[23118] {pull}23931[23931] -- Updated aws/cloudtrail fileset to ECS 1.8. {issue}23118[23118] {pull}23911[23911] - Upgrade juniper/srx to ecs 1.8.0. {issue}23118[23118] {pull}23936[23936] -- Update mysqlenterprise module to ECS 1.8. {issue}23118[23118] {pull}23978[23978] -- Upgrade sophos/xg fileset to ECS 1.8.0. {issue}23118[23118] {pull}23967[23967] -- Upgrade system/auth to ECS 1.8 {issue}23118[23118] {pull}23961[23961] -- Upgrade elasticsearch/audit to ECS 1.8 {issue}23118[23118] {pull}24000[24000] - Upgrade okta to ecs 1.8.0 and move js processor to ingest pipeline {issue}23118[23118] {pull}23929[23929] -- Update zoom module to ECS 1.8. {pull}23904[23904] {issue}23118[23118] -- Support X-Forwarder-For in IIS logs. {pull}19142[192142] -- Add support for logs generated by servers configured with `log_statement` and `log_duration` in PostgreSQL module. {pull}24607[24607] -- Updating field mappings for Cisco AMP module, fixing certain fields. {pull}24661[24661] -- Added fifteen new message IDs to Cisco ASA/FTD pipeline. {pull}24744[24744] -- Added NTP fileset to Zeek module {pull}24224[24224] -- Add `proxy_url` config for httpjson v2 input. {issue}24615[24615] {pull}24662[24662] -- Add `fail_on_template_error` option for httpjson input. {pull}24784[24784] -- Change `okta.target` to `flattened` field type. {issue}24354[24354] {pull}24636[24636] -- Added `http.request.id` to `nginx/ingress_controller` and `elasticsearch/audit`. {pull}24994[24994] -- Add `awsfargate` module to collect container logs from Amazon ECS on Fargate. {pull}25041[25041] -- New module `cyberarkpas` for CyberArk Privileged Access Security audit logs. {pull}24803[24803] -- Add `uri_parts` processor to Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules ingest pipelines. {issue}19088[19088] {pull}24699[24699] -- New module `zookeeper` for Zookeeper service and audit logs {issue}25061[25061] {pull}25128[25128] -- Add parsing for `haproxy.http.request.raw_request_line` field {issue}25480[25480] {pull}25482[25482] -- Mark `filestream` input beta. {pull}25560[25560] *Heartbeat* -- Add mime type detection for http responses. {pull}22976[22976] - Bundle synthetics deps with heartbeat docker image. {pull}23274[23274] -- Handle datastreams for fleet. {pull}24223[24223] -- Add --sandbox option for browser monitor. {pull}24172[24172] -- Support additional 'root' fields from synthetics. {pull}24770[24770] -- Browser zip_url source type. {pull}24714[24714] *Heartbeat* @@ -650,29 +435,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Add billing metricset into googlecloud module. {pull}20812[20812] {issue}20738[20738] - Release lambda metricset in aws module as GA. {issue}21251[21251] {pull}21255[21255] - Add dashboard for pubsub metricset in googlecloud module. {pull}21326[21326] {issue}17137[17137] -- Move Prometheus query & remote_write to GA. {pull}21507[21507] -- Map cloud data filed `cloud.account.id` to azure subscription. {pull}21483[21483] {issue}21381[21381] -- Expand unsupported option from namespace to metrics in the azure module. {pull}21486[21486] -- Move s3_daily_storage and s3_request metricsets to use cloudwatch input. {pull}21703[21703] -- Duplicate system.process.cmdline field with process.command_line ECS field name. {pull}22325[22325] -- Add awsfargate module task_stats metricset to monitor AWS ECS Fargate. {pull}22034[22034] -- Add connection and route metricsets for nats metricbeat module to collect metrics per connection/route. {pull}22445[22445] -- Add unit file states to system/service {pull}22557[22557] -- `kibana` module: `stats` metricset no-longer collects usage-related data. {pull}22732[22732] -- Add more TCP states to Metricbeat system socket_summary. {pull}14347[14347] -- Add io.ops in fields exported by system.diskio. {pull}22066[22066] -- Adjust the Apache status fields in the fleet mode. {pull}22821[22821] -- Add AWS Fargate overview dashboard. {pull}22941[22941] -- Add process.state, process.cpu.pct, process.cpu.start_time and process.memory.pct. {pull}22845[22845] -- Move IIS module to GA and map fields. {issue}22609[22609] {pull}23024[23024] -- Apache: convert status.total_kbytes to status.total_bytes in fleet mode. {pull}23022[23022] -- Release MSSQL as GA {pull}23146[23146] - Enrich events of `state_service` metricset with kubernetes services' metadata. {pull}23730[23730] -- Add support for Darwin/arm M1. {pull}24019[24019] - Check fields are documented in aws metricsets. {pull}23887[23887] -- Add support for defining metrics_filters for prometheus module in hints. {pull}24264[24264] -- Add support for PostgreSQL 10, 11, 12 and 13. {pull}24402[24402] -- Add support for SASL/SCRAM authentication to the Kafka module. {pull}24810[24810] *Packetbeat* @@ -689,7 +453,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Set process.command_line and process.parent.command_line from Sysmon Event ID 1. {pull}17327[17327] - Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module {pull}17517[17517] - Add registry and code signature information and ECS categorization fields for sysmon module {pull}18058[18058] -- Add support for sysmon v13 events 24 and 25. {issue}24217[24217] {pull}24945[24945] *Elastic Log Driver* diff --git a/libbeat/docs/release.asciidoc b/libbeat/docs/release.asciidoc index a53bf859bc3f..530506f4a070 100644 --- a/libbeat/docs/release.asciidoc +++ b/libbeat/docs/release.asciidoc @@ -8,6 +8,7 @@ This section summarizes the changes in each release. Also read <> for more detail about changes that affect upgrade. +* <> * <> * <> * <>