docs: Prepare Changelog for 8.11.0 (#37058)

* docs: Close changelog for 8.11.0 * Update CHANGELOG.asciidoc * Update CHANGELOG.asciidoc * Update CHANGELOG.next.asciidoc * Apply suggestions from code review Co-authored-by: David Kilfoyle <[email protected]> --------- Co-authored-by: Pierre HILBERT <[email protected]> Co-authored-by: David Kilfoyle <[email protected]> (cherry picked from commit 0308d29)
elastic · Nov 7, 2023 · 8b57c9b · 8b57c9b
1 parent 04b3989
commit 8b57c9b
Show file tree

Hide file tree

Showing 3 changed files with 91 additions and 65 deletions.
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
@@ -3,6 +3,91 @@
 :issue: https://github.com/elastic/beats/issues/
 :pull: https://github.com/elastic/beats/pull/
 
+[[release-notes-8.11.0]]
+=== Beats version 8.11.0
+https://github.com/elastic/beats/compare/v8.10.4\...v8.11.0[View commits]
+
+==== Breaking changes
+
+*Affecting all Beats*
+- The Elasticsearch output now enables compression by default. This decreases network data usage by an average of 70-80%, in exchange for 20-25% increased CPU use and ~10% increased ingestion time. The previous default can be restored by setting the flag `compression_level: 0` under `output.elasticsearch`. {pull}36681[36681]
+- The `elastic-agent-autodiscover` library is updated to version 0.6.4, disabling metadata for deployment and cronjob. Pods that will be created from deployments or cronjobs will not have the extra metadata field for `kubernetes.deployment` or `kubernetes.cronjob`, respectively. {pull}36879[36879]
+
+*Filebeat*
+
+- Switch types of `log.file.device`, `log.file.inode`, `log.file.idxhi`, `log.file.idxlo` and `log.file.vol` fields to strings to better align with ECS and integrations. {pull}36697[36697]
+
+*Metricbeat*
+
+- The System module now collects the number of threads per process. The elastic-agent-system-metrics was updated to v0.7.0 as this version collects the number of threads.
+
+==== Bugfixes
+
+*Affecting all Beats*
+
+- Upgrade `elastic-agent-libs` to v0.6.0, allowing a Beat running as a Windows service to receive more than one change request. {pull}36896[36896]
+
+*Filebeat*
+
+- Added a fix for the Crowdstrike pipeline handling of process arrays. {pull}36496[36496]
+- Fix handling of response errors in HTTPJSON and CEL request trace logging. {pull}36956[36956]
+
+*Heartbeat*
+
+- Fix retries to trigger on a down monitor with no previous state. {pull}36842[36842]
+- Bump NodeJS minor version to 18.18.2. {pull}36961[36961]
+- Fix monitor duration calculation with retries. {pull}36900[36900]
+
+*Metricbeat*
+
+- Fix Azure Monitor empty metricnamespace. {pull}36295[36295]
+- Fix GCP compute metadata. {pull}36338[36338]
+- Add missing 'TransactionType' dimension for Azure Storage Account. {pull}36413[36413]
+- Add log error when statsd server fails to start. {pull}36477[36477]
+- Fix CassandraConnectionClosures metric configuration. {pull}34742[34742]
+- Fix event mapping implementation for statsd module. {pull}36925[36925]
+
+*Winlogbeat*
+
+- Fix User Account Control Attributes Table values for Security module. {issue}36999[36999] {pull}37009[37009]
+
+==== Added
+
+*Affecting all Beats*
+
+- Upgrade to Go 1.20.10. {pull}36846[36846]
+- Add support for forward lookups (`A`, `AAAA`, and `TXT`) in DNS processor. {issue}11416[11416] {pull}36394[36394]
+- Mark `syslog` processor as GA, improve docs about how processor handles syslog messages. {issue}36416[36416] {pull}36417[36417]
+- Add support for AWS external IDs. {issue}36321[36321] {pull}36322[36322]
+- Disable `netinfo.enabled` option of `add-host-metadata processor`to enhance `host.ip` and `host.mac`. {pull}36506[36506]
+- {Beats} will now connect to older {es} instances by default. {pull}36884[36884]
+- Upgrade golang/x/net to v0.17.0. Updates the publicsuffix table used by the registered_domain processor. {pull}36969[36969]
+
+*Filebeat*
+
+- Reduce HTTPJSON metrics allocations. {pull}36282[36282]
+- Add support for a simplified input configuraton when running under {agent}. {pull}36390[36390]
+- Make HTTPJSON response body decoding errors more informative. {pull}36481[36481]
+- Allow fine-grained control of entity analytics API requests for Okta provider. {issue}36440[36440] {pull}36492[36492]
+- Add support for expanding `journald.process.capabilities` into the human-readable effective capabilities in the ECS `process.thread.capabilities.effective` field. {issue}36454[36454] {pull}36470[36470]
+- Allow fine-grained control of entity analytics API requests for AzureAD provider. {issue}36440[36440] {pull}36441[36441]
+- For request tracer logging in CEL and httpjson the request and response body are no longer included in `event.original`. The body is still present in `http.{request,response}.body.content`. {pull}36531[36531]
+- Add support for Okta OAuth2 provider in the CEL input. {issue}36336[36336] {pull}36521[36521]
+- Improve error logging in HTTPJSON input. {pull}36529[36529]
+- Disable warning message about ingest pipeline loading when running under Elastic Agent. {pull}36659[36659]
+- Add input metrics to http_endpoint input. {issue}36402[36402] {pull}36427[36427]
+- Remove Event Normalization from GCP PubSub Input. {pull}36716[36716]
+- Add support for new features & removed partial save mechanism in the Azure Blob Storage input. {issue}35126[35126] {pull}36690[36690]
+- Improve template evaluation logging for HTTPJSON input. {pull}36668[36668]
+- Add CEL partial value debug function. {pull}36652[36652]
+- Add support for new features and remove partial save mechanism in the GCS input. {issue}35847[35847] {pull}36713[36713]
+- Add cache processor. {pull}36786[36786]
+
+*Packetbeat*
+
+- Bump Windows Npcap version to v1.76. {issue}36539[36539] {pull}36549[36549]
+
+
 [[release-notes-8.10.4]]
 === Beats version 8.10.4
 https://github.com/elastic/beats/compare/v8.10.3\...v8.10.4[View commits]

diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -24,16 +24,12 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 
 *Filebeat*
 
-- Switch types of `log.file.device`, `log.file.inode`, `log.file.idxhi`, `log.file.idxlo` and `log.file.vol` fields to strings to better align with ECS and integrations. {pull}36697[36697]
 
 *Heartbeat*
 - Decreases the ES default timeout to 10 for the load monitor state requests
 
 *Metricbeat*
 
- - System module now collects the number of threads per process.
-The elastic-agent-system-metrics was updated to v0.7.0 as the number of threads
-is collected by it.
 
 *Osquerybeat*
 
@@ -64,16 +60,7 @@ is collected by it.
 - 'add_cloud_metadata' processor - update azure metadata api version to get missing `cloud.account.id` field
 - Upgraded apache arrow library used in x-pack/libbeat/reader/parquet from v11 to v12.0.1 in order to fix cross-compilation issues {pull}35640[35640]
 - Fix panic when MaxRetryInterval is specified, but RetryInterval is not {pull}35820[35820]
-- Do not print context cancelled error message when running under agent {pull}36006[36006]
-- Fix recovering from invalid output configuration when running under Elastic-Agent {pull}36016[36016]
-- Improve StreamBuf append to improve performance when reading long lines from files. {pull}35928[35928]
-- Eliminate cloning of event in deepUpdate {pull}35945[35945]
-- Fix ndjson parser to store JSON fields correctly under `target` {issue}29395[29395]
 - Support build of projects outside of beats directory {pull}36126[36126]
-- Add default cgroup regex for add_process_metadata processor {pull}36484[36484] {issue}32961[32961]
-- Fix environment capture by `add_process_metadata` processor. {issue}36469[36469] {pull}36471[36471]
-- syslog processor - Fix the ability to use `when` conditions on the processor. {issue}36762[36762]
-- upgrade elastic-agent-libs to v0.6.0, allows beat running as a windows service to receive more than one change request. {pull}36896[36896]
 
 *Auditbeat*
 
@@ -87,28 +74,9 @@ is collected by it.
 - Fixes "Can only start an input when all related states are finished" error when running under Elastic-Agent {pull}35250[35250] {issue}33653[33653]
 - [system] sync system/auth dataset with system integration 1.29.0. {pull}35581[35581]
 - [GCS Input] - Fixed an issue where bucket_timeout was being applied to the entire bucket poll interval and not individual bucket object read operations. Fixed a map write concurrency issue arising from data races when using a high number of workers. Fixed the flaky tests that were present in the GCS test suit. {pull}35605[35605]
-- Fix filestream false positive log error "filestream input with ID 'xyz' already exists" {issue}31767[31767]
-- Fix error message formatting from filestream input. {pull}35658[35658]
-- Fix error when trying to use `include_message` parser {issue}35440[35440]
-- Fix handling of IPv6 unspecified addresses in TCP input. {issue}35064[35064] {pull}35637[35637]
-- Fixed a minor code error in the GCS input scheduler where a config value was being used directly instead of the source struct. {pull}35729[35729]
-- Improve error reporting and fix IPv6 handling of TCP and UDP metric collection. {pull}35772[35772]
-- Fix CEL input JSON marshalling of nested objects. {issue}35763[35763] {pull}35774[35774]
-- Fix metric collection in GCPPubSub input. {pull}35773[35773]
-- Fix end point deregistration in http_endpoint input. {issue}35899[35899] {pull}35903[35903]
-- Fix duplicate ID panic in filestream metrics. {issue}35964[35964] {pull}35972[35972]
-- Improve error reporting and fix IPv6 handling of TCP and UDP metric collection. {pull}35996[35996]
-- Fix handling of NUL-terminated log lines in Fortinet Firewall module. {issue}36026[36026] {pull}36027[36027]
-- Make redact field configuration recommended in CEL input and log warning if missing. {pull}36008[36008]
-- Fix handling of region name configuration in awss3 input {pull}36034[36034]
 - Fixed concurrency and flakey tests issue in azure blob storage input. {issue}35983[35983] {pull}36124[36124]
 - Fix panic when sqs input metrics getter is invoked {pull}36101[36101] {issue}36077[36077]
-- Make CEL input's `now` global variable static for evaluation lifetime. {pull}36107[36107]
-- Update mito CEL extension library to v1.5.0. {pull}36146[36146]
-- Filter out duplicate paths resolved from matching globs. {issue}36253[36253] {pull}36256[36256]
-- Fix handling of TCP/UDP address resolution during metric initialization. {issue}35064[35064] {pull}36287[36287]
 - Fix handling of Juniper SRX structured data when there is no leading junos element. {issue}36270[36270] {pull}36308[36308]
-- Remove erroneous error log in GCPPubSub input. {pull}36296[36296]
 - Fix Filebeat Cisco module with missing escape character {issue}36325[36325] {pull}36326[36326]
 - Fix panic when redact option is not provided to CEL input. {issue}36387[36387] {pull}36388[36388]
 - Remove 'onFilteredOut' and 'onDroppedOnPublish' callback logs {issue}36299[36299] {pull}36399[36399]
@@ -123,9 +91,6 @@ is collected by it.
 *Heartbeat*
 
 - Fix panics when parsing dereferencing invalid parsed url. {pull}34702[34702]
-- Fix retries to trigger on a down monitor with no previous state. {pull}36842[36842]
-- Bump NodeJS minor version to 18.18.2. {pull}36961[36961]
-- Fix monitor duration calculation with retries. {pull}36900[36900]
 
 *Metricbeat*
 
@@ -139,12 +104,7 @@ is collected by it.
 - Collect missing remote_cluster in elasticsearch ccr metricset {pull}34957[34957]
 - Add context with timeout in AWS API calls {pull}35425[35425]
 - Fix EC2 host.cpu.usage {pull}35717[35717]
-- Resolve statsd module's prematurely halting of metrics parsing upon encountering an invalid packet. {pull}35075[35075]
-- Fix the gap in fetching forecast API metrics at the end of each month for Azure billing module  {pull}36142[36142]
 - Add option in SQL module to execute queries for all dbs. {pull}35688[35688]
-- Fix Azure Monitor empty metricnamespace. {pull}36295[36295]
-- Fix GCP compute metadata. {pull}36338[36338]
-- Add support for api_key authentication in elasticsearch module  {pull}36274[36274]
 - Add remaining dimensions for azure storage account to make them available for tsdb enablement. {pull}36331[36331]
 - Add missing 'TransactionType' dimension for Azure Storage Account. {pull}36413[36413]
 - Add log error when statsd server fails to start {pull}36477[36477]
@@ -160,7 +120,6 @@ is collected by it.
 
 *Winlogbeat*
 
-- Fix User Account Control Attributes Table values for Security module. {issue}36999[36999] {pull}37009[37009]
 
 *Elastic Logging Plugin*
 
@@ -169,7 +128,6 @@ is collected by it.
 
 *Affecting all Beats*
 
-- Upgrade to Go 1.20.10. {pull}36846[36846]
 - Added append Processor which will append concrete values or values from a field to target. {issue}29934[29934] {pull}33364[33364]
 - When running under Elastic-Agent the status is now reported per Unit instead of the whole Beat {issue}35874[35874] {pull}36183[36183]
 - Add warning message to SysV init scripts for RPM-based systems that lack `/etc/rc.d/init.d/functions`. {issue}35708[35708] {pull}36188[36188]
@@ -184,6 +142,7 @@ is collected by it.
 - Raise up logging level to warning when attempting to configure beats with unknown fields from autodiscovered events/environments
 - elasticsearch output now supports `idle_connection_timeout`. {issue}35616[35615] {pull}36843[36843]
 - Upgrade golang/x/net to v0.17.0. Updates the publicsuffix table used by the registered_domain processor. {pull}36969[36969]
+Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will disable the netinfo.enabled option of add_host_metadata processor
 
 *Auditbeat*
 
@@ -204,27 +163,6 @@ is collected by it.
 - Add MySQL authentication message parsing and `related.ip` and `related.user` fields {pull}34810[34810]
 - Add nginx ingress_controller parsing if one of upstreams fails to return response {pull}34787[34787]
 - Add oracle authentication messages parsing {pull}35127[35127]
-- Add sanitization capabilities to azure-eventhub input {pull}34874[34874]
-- Add support for CRC validation in Filebeat's HTTP endpoint input. {pull}35204[35204]
-- Add support for CRC validation in Zoom module. {pull}35604[35604]
-- Add execution budget to CEL input. {pull}35409[35409]
-- Add XML decoding support to HTTPJSON. {issue}34438[34438] {pull}35235[35235]
-- Add delegated account support when using Google ADC in `httpjson` input. {pull}35507[35507]
-- Allow specifying since when to read journald entries. {pull}35408[35408]
-- Add metrics for filestream input. {pull}35529[35529]
-- Add support for collecting `httpjson` metrics. {pull}35392[35392]
-- Add XML decoding support to CEL. {issue}34438[34438] {pull}35372[35372]
-- Mark CEL input as GA. {pull}35559[35559]
-- Add metrics for gcp-pubsub input. {pull}35614[35614]
-- [GCS] Added scheduler debug logs and improved the context passing mechanism by removing them from struct params and passing them as function arguments. {pull}35674[35674]
-- Allow non-AWS endpoints for awss3 input. {issue}35496[35496] {pull}35520[35520]
-- Under elastic-agent the input metrics will now be included in agent diagnostics dumps. {pull}35798[35798]
-- Add Okta input package for entity analytics. {pull}35611[35611]
-- Expose harvester metrics from filestream input {pull}35835[35835] {issue}33771[33771]
-- Add device support for Azure AD entity analytics. {pull}35807[35807]
-- Improve CEL input performance. {pull}35915[35915]
-- Adding filename details from zip to response for httpjson {issue}33952[33952] {pull}34044[34044]
-- Added support for min/max template functions in httpjson input. {issue}36094[36094] {pull}36036[36036]
 - Add `clean_session` configuration setting for MQTT input.  {pull}35806[16204]
 - Add fingerprint mode for the filestream scanner and new file identity based on it {issue}34419[34419] {pull}35734[35734]
 - Add file system metadata to events ingested via filestream {issue}35801[35801] {pull}36065[36065]
@@ -262,6 +200,7 @@ is collected by it.
 - Add support for Digest Authentication to CEL input. {issue}35514[35514] {pull}36932[36932]
 - Use filestream input with file_identity.fingerprint as default for hints autodiscover. {issue}35984[35984] {pull}36950[36950]
 - Add network processor in addition to interface based direction resolution. {pull}37023[37023]
+- Add setup option `--force-enable-module-filesets`, that will act as if all filesets have been enabled in a module during setup. {issue}30915[30915] {pull}99999[99999]
 
 *Auditbeat*
 
@@ -291,8 +230,6 @@ is collected by it.
 
 *Packetbeat*
 
-- Improve efficiency of sniffers by deduplicating interface configurations. {issue}36574[36574] {pull}36576[36576]
-- Bump Windows Npcap version to v1.76. {issue}36539[36539] {pull}36549[36549]
 
 *Packetbeat*
 
@@ -364,6 +301,9 @@ is collected by it.
 
 
 
+
+
+
 
 
 

diff --git a/libbeat/docs/release.asciidoc b/libbeat/docs/release.asciidoc
@@ -8,6 +8,7 @@ This section summarizes the changes in each release. Also read
 <<breaking-changes>> for more detail about changes that affect
 upgrade.
 
+* <<release-notes-8.11.0>>
 * <<release-notes-8.10.4>>
 * <<release-notes-8.10.3>>
 * <<release-notes-8.10.2>>