From 8b57c9b5be18938411971b401fd53ced189446d3 Mon Sep 17 00:00:00 2001 From: Elastic Machine Date: Wed, 8 Nov 2023 04:34:03 +1100 Subject: [PATCH] docs: Prepare Changelog for 8.11.0 (#37058) * docs: Close changelog for 8.11.0 * Update CHANGELOG.asciidoc * Update CHANGELOG.asciidoc * Update CHANGELOG.next.asciidoc * Apply suggestions from code review Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> --------- Co-authored-by: Pierre HILBERT Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> (cherry picked from commit 0308d2972a8629f3598656fdaedf2f670c22a1fc) --- CHANGELOG.asciidoc | 85 +++++++++++++++++++++++++++++++++++ CHANGELOG.next.asciidoc | 70 +++-------------------------- libbeat/docs/release.asciidoc | 1 + 3 files changed, 91 insertions(+), 65 deletions(-) diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index e97fd625570..e3dd7eb96d6 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -3,6 +3,91 @@ :issue: https://github.com/elastic/beats/issues/ :pull: https://github.com/elastic/beats/pull/ +[[release-notes-8.11.0]] +=== Beats version 8.11.0 +https://github.com/elastic/beats/compare/v8.10.4\...v8.11.0[View commits] + +==== Breaking changes + +*Affecting all Beats* +- The Elasticsearch output now enables compression by default. This decreases network data usage by an average of 70-80%, in exchange for 20-25% increased CPU use and ~10% increased ingestion time. The previous default can be restored by setting the flag `compression_level: 0` under `output.elasticsearch`. {pull}36681[36681] +- The `elastic-agent-autodiscover` library is updated to version 0.6.4, disabling metadata for deployment and cronjob. Pods that will be created from deployments or cronjobs will not have the extra metadata field for `kubernetes.deployment` or `kubernetes.cronjob`, respectively. {pull}36879[36879] + +*Filebeat* + +- Switch types of `log.file.device`, `log.file.inode`, `log.file.idxhi`, `log.file.idxlo` and `log.file.vol` fields to strings to better align with ECS and integrations. {pull}36697[36697] + +*Metricbeat* + +- The System module now collects the number of threads per process. The elastic-agent-system-metrics was updated to v0.7.0 as this version collects the number of threads. + +==== Bugfixes + +*Affecting all Beats* + +- Upgrade `elastic-agent-libs` to v0.6.0, allowing a Beat running as a Windows service to receive more than one change request. {pull}36896[36896] + +*Filebeat* + +- Added a fix for the Crowdstrike pipeline handling of process arrays. {pull}36496[36496] +- Fix handling of response errors in HTTPJSON and CEL request trace logging. {pull}36956[36956] + +*Heartbeat* + +- Fix retries to trigger on a down monitor with no previous state. {pull}36842[36842] +- Bump NodeJS minor version to 18.18.2. {pull}36961[36961] +- Fix monitor duration calculation with retries. {pull}36900[36900] + +*Metricbeat* + +- Fix Azure Monitor empty metricnamespace. {pull}36295[36295] +- Fix GCP compute metadata. {pull}36338[36338] +- Add missing 'TransactionType' dimension for Azure Storage Account. {pull}36413[36413] +- Add log error when statsd server fails to start. {pull}36477[36477] +- Fix CassandraConnectionClosures metric configuration. {pull}34742[34742] +- Fix event mapping implementation for statsd module. {pull}36925[36925] + +*Winlogbeat* + +- Fix User Account Control Attributes Table values for Security module. {issue}36999[36999] {pull}37009[37009] + +==== Added + +*Affecting all Beats* + +- Upgrade to Go 1.20.10. {pull}36846[36846] +- Add support for forward lookups (`A`, `AAAA`, and `TXT`) in DNS processor. {issue}11416[11416] {pull}36394[36394] +- Mark `syslog` processor as GA, improve docs about how processor handles syslog messages. {issue}36416[36416] {pull}36417[36417] +- Add support for AWS external IDs. {issue}36321[36321] {pull}36322[36322] +- Disable `netinfo.enabled` option of `add-host-metadata processor`to enhance `host.ip` and `host.mac`. {pull}36506[36506] +- {Beats} will now connect to older {es} instances by default. {pull}36884[36884] +- Upgrade golang/x/net to v0.17.0. Updates the publicsuffix table used by the registered_domain processor. {pull}36969[36969] + +*Filebeat* + +- Reduce HTTPJSON metrics allocations. {pull}36282[36282] +- Add support for a simplified input configuraton when running under {agent}. {pull}36390[36390] +- Make HTTPJSON response body decoding errors more informative. {pull}36481[36481] +- Allow fine-grained control of entity analytics API requests for Okta provider. {issue}36440[36440] {pull}36492[36492] +- Add support for expanding `journald.process.capabilities` into the human-readable effective capabilities in the ECS `process.thread.capabilities.effective` field. {issue}36454[36454] {pull}36470[36470] +- Allow fine-grained control of entity analytics API requests for AzureAD provider. {issue}36440[36440] {pull}36441[36441] +- For request tracer logging in CEL and httpjson the request and response body are no longer included in `event.original`. The body is still present in `http.{request,response}.body.content`. {pull}36531[36531] +- Add support for Okta OAuth2 provider in the CEL input. {issue}36336[36336] {pull}36521[36521] +- Improve error logging in HTTPJSON input. {pull}36529[36529] +- Disable warning message about ingest pipeline loading when running under Elastic Agent. {pull}36659[36659] +- Add input metrics to http_endpoint input. {issue}36402[36402] {pull}36427[36427] +- Remove Event Normalization from GCP PubSub Input. {pull}36716[36716] +- Add support for new features & removed partial save mechanism in the Azure Blob Storage input. {issue}35126[35126] {pull}36690[36690] +- Improve template evaluation logging for HTTPJSON input. {pull}36668[36668] +- Add CEL partial value debug function. {pull}36652[36652] +- Add support for new features and remove partial save mechanism in the GCS input. {issue}35847[35847] {pull}36713[36713] +- Add cache processor. {pull}36786[36786] + +*Packetbeat* + +- Bump Windows Npcap version to v1.76. {issue}36539[36539] {pull}36549[36549] + + [[release-notes-8.10.4]] === Beats version 8.10.4 https://github.com/elastic/beats/compare/v8.10.3\...v8.10.4[View commits] diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 393a263e7c8..462af8725a1 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -24,16 +24,12 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff] *Filebeat* -- Switch types of `log.file.device`, `log.file.inode`, `log.file.idxhi`, `log.file.idxlo` and `log.file.vol` fields to strings to better align with ECS and integrations. {pull}36697[36697] *Heartbeat* - Decreases the ES default timeout to 10 for the load monitor state requests *Metricbeat* - - System module now collects the number of threads per process. -The elastic-agent-system-metrics was updated to v0.7.0 as the number of threads -is collected by it. *Osquerybeat* @@ -64,16 +60,7 @@ is collected by it. - 'add_cloud_metadata' processor - update azure metadata api version to get missing `cloud.account.id` field - Upgraded apache arrow library used in x-pack/libbeat/reader/parquet from v11 to v12.0.1 in order to fix cross-compilation issues {pull}35640[35640] - Fix panic when MaxRetryInterval is specified, but RetryInterval is not {pull}35820[35820] -- Do not print context cancelled error message when running under agent {pull}36006[36006] -- Fix recovering from invalid output configuration when running under Elastic-Agent {pull}36016[36016] -- Improve StreamBuf append to improve performance when reading long lines from files. {pull}35928[35928] -- Eliminate cloning of event in deepUpdate {pull}35945[35945] -- Fix ndjson parser to store JSON fields correctly under `target` {issue}29395[29395] - Support build of projects outside of beats directory {pull}36126[36126] -- Add default cgroup regex for add_process_metadata processor {pull}36484[36484] {issue}32961[32961] -- Fix environment capture by `add_process_metadata` processor. {issue}36469[36469] {pull}36471[36471] -- syslog processor - Fix the ability to use `when` conditions on the processor. {issue}36762[36762] -- upgrade elastic-agent-libs to v0.6.0, allows beat running as a windows service to receive more than one change request. {pull}36896[36896] *Auditbeat* @@ -87,28 +74,9 @@ is collected by it. - Fixes "Can only start an input when all related states are finished" error when running under Elastic-Agent {pull}35250[35250] {issue}33653[33653] - [system] sync system/auth dataset with system integration 1.29.0. {pull}35581[35581] - [GCS Input] - Fixed an issue where bucket_timeout was being applied to the entire bucket poll interval and not individual bucket object read operations. Fixed a map write concurrency issue arising from data races when using a high number of workers. Fixed the flaky tests that were present in the GCS test suit. {pull}35605[35605] -- Fix filestream false positive log error "filestream input with ID 'xyz' already exists" {issue}31767[31767] -- Fix error message formatting from filestream input. {pull}35658[35658] -- Fix error when trying to use `include_message` parser {issue}35440[35440] -- Fix handling of IPv6 unspecified addresses in TCP input. {issue}35064[35064] {pull}35637[35637] -- Fixed a minor code error in the GCS input scheduler where a config value was being used directly instead of the source struct. {pull}35729[35729] -- Improve error reporting and fix IPv6 handling of TCP and UDP metric collection. {pull}35772[35772] -- Fix CEL input JSON marshalling of nested objects. {issue}35763[35763] {pull}35774[35774] -- Fix metric collection in GCPPubSub input. {pull}35773[35773] -- Fix end point deregistration in http_endpoint input. {issue}35899[35899] {pull}35903[35903] -- Fix duplicate ID panic in filestream metrics. {issue}35964[35964] {pull}35972[35972] -- Improve error reporting and fix IPv6 handling of TCP and UDP metric collection. {pull}35996[35996] -- Fix handling of NUL-terminated log lines in Fortinet Firewall module. {issue}36026[36026] {pull}36027[36027] -- Make redact field configuration recommended in CEL input and log warning if missing. {pull}36008[36008] -- Fix handling of region name configuration in awss3 input {pull}36034[36034] - Fixed concurrency and flakey tests issue in azure blob storage input. {issue}35983[35983] {pull}36124[36124] - Fix panic when sqs input metrics getter is invoked {pull}36101[36101] {issue}36077[36077] -- Make CEL input's `now` global variable static for evaluation lifetime. {pull}36107[36107] -- Update mito CEL extension library to v1.5.0. {pull}36146[36146] -- Filter out duplicate paths resolved from matching globs. {issue}36253[36253] {pull}36256[36256] -- Fix handling of TCP/UDP address resolution during metric initialization. {issue}35064[35064] {pull}36287[36287] - Fix handling of Juniper SRX structured data when there is no leading junos element. {issue}36270[36270] {pull}36308[36308] -- Remove erroneous error log in GCPPubSub input. {pull}36296[36296] - Fix Filebeat Cisco module with missing escape character {issue}36325[36325] {pull}36326[36326] - Fix panic when redact option is not provided to CEL input. {issue}36387[36387] {pull}36388[36388] - Remove 'onFilteredOut' and 'onDroppedOnPublish' callback logs {issue}36299[36299] {pull}36399[36399] @@ -123,9 +91,6 @@ is collected by it. *Heartbeat* - Fix panics when parsing dereferencing invalid parsed url. {pull}34702[34702] -- Fix retries to trigger on a down monitor with no previous state. {pull}36842[36842] -- Bump NodeJS minor version to 18.18.2. {pull}36961[36961] -- Fix monitor duration calculation with retries. {pull}36900[36900] *Metricbeat* @@ -139,12 +104,7 @@ is collected by it. - Collect missing remote_cluster in elasticsearch ccr metricset {pull}34957[34957] - Add context with timeout in AWS API calls {pull}35425[35425] - Fix EC2 host.cpu.usage {pull}35717[35717] -- Resolve statsd module's prematurely halting of metrics parsing upon encountering an invalid packet. {pull}35075[35075] -- Fix the gap in fetching forecast API metrics at the end of each month for Azure billing module {pull}36142[36142] - Add option in SQL module to execute queries for all dbs. {pull}35688[35688] -- Fix Azure Monitor empty metricnamespace. {pull}36295[36295] -- Fix GCP compute metadata. {pull}36338[36338] -- Add support for api_key authentication in elasticsearch module {pull}36274[36274] - Add remaining dimensions for azure storage account to make them available for tsdb enablement. {pull}36331[36331] - Add missing 'TransactionType' dimension for Azure Storage Account. {pull}36413[36413] - Add log error when statsd server fails to start {pull}36477[36477] @@ -160,7 +120,6 @@ is collected by it. *Winlogbeat* -- Fix User Account Control Attributes Table values for Security module. {issue}36999[36999] {pull}37009[37009] *Elastic Logging Plugin* @@ -169,7 +128,6 @@ is collected by it. *Affecting all Beats* -- Upgrade to Go 1.20.10. {pull}36846[36846] - Added append Processor which will append concrete values or values from a field to target. {issue}29934[29934] {pull}33364[33364] - When running under Elastic-Agent the status is now reported per Unit instead of the whole Beat {issue}35874[35874] {pull}36183[36183] - Add warning message to SysV init scripts for RPM-based systems that lack `/etc/rc.d/init.d/functions`. {issue}35708[35708] {pull}36188[36188] @@ -184,6 +142,7 @@ is collected by it. - Raise up logging level to warning when attempting to configure beats with unknown fields from autodiscovered events/environments - elasticsearch output now supports `idle_connection_timeout`. {issue}35616[35615] {pull}36843[36843] - Upgrade golang/x/net to v0.17.0. Updates the publicsuffix table used by the registered_domain processor. {pull}36969[36969] +Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will disable the netinfo.enabled option of add_host_metadata processor *Auditbeat* @@ -204,27 +163,6 @@ is collected by it. - Add MySQL authentication message parsing and `related.ip` and `related.user` fields {pull}34810[34810] - Add nginx ingress_controller parsing if one of upstreams fails to return response {pull}34787[34787] - Add oracle authentication messages parsing {pull}35127[35127] -- Add sanitization capabilities to azure-eventhub input {pull}34874[34874] -- Add support for CRC validation in Filebeat's HTTP endpoint input. {pull}35204[35204] -- Add support for CRC validation in Zoom module. {pull}35604[35604] -- Add execution budget to CEL input. {pull}35409[35409] -- Add XML decoding support to HTTPJSON. {issue}34438[34438] {pull}35235[35235] -- Add delegated account support when using Google ADC in `httpjson` input. {pull}35507[35507] -- Allow specifying since when to read journald entries. {pull}35408[35408] -- Add metrics for filestream input. {pull}35529[35529] -- Add support for collecting `httpjson` metrics. {pull}35392[35392] -- Add XML decoding support to CEL. {issue}34438[34438] {pull}35372[35372] -- Mark CEL input as GA. {pull}35559[35559] -- Add metrics for gcp-pubsub input. {pull}35614[35614] -- [GCS] Added scheduler debug logs and improved the context passing mechanism by removing them from struct params and passing them as function arguments. {pull}35674[35674] -- Allow non-AWS endpoints for awss3 input. {issue}35496[35496] {pull}35520[35520] -- Under elastic-agent the input metrics will now be included in agent diagnostics dumps. {pull}35798[35798] -- Add Okta input package for entity analytics. {pull}35611[35611] -- Expose harvester metrics from filestream input {pull}35835[35835] {issue}33771[33771] -- Add device support for Azure AD entity analytics. {pull}35807[35807] -- Improve CEL input performance. {pull}35915[35915] -- Adding filename details from zip to response for httpjson {issue}33952[33952] {pull}34044[34044] -- Added support for min/max template functions in httpjson input. {issue}36094[36094] {pull}36036[36036] - Add `clean_session` configuration setting for MQTT input. {pull}35806[16204] - Add fingerprint mode for the filestream scanner and new file identity based on it {issue}34419[34419] {pull}35734[35734] - Add file system metadata to events ingested via filestream {issue}35801[35801] {pull}36065[36065] @@ -262,6 +200,7 @@ is collected by it. - Add support for Digest Authentication to CEL input. {issue}35514[35514] {pull}36932[36932] - Use filestream input with file_identity.fingerprint as default for hints autodiscover. {issue}35984[35984] {pull}36950[36950] - Add network processor in addition to interface based direction resolution. {pull}37023[37023] +- Add setup option `--force-enable-module-filesets`, that will act as if all filesets have been enabled in a module during setup. {issue}30915[30915] {pull}99999[99999] *Auditbeat* @@ -291,8 +230,6 @@ is collected by it. *Packetbeat* -- Improve efficiency of sniffers by deduplicating interface configurations. {issue}36574[36574] {pull}36576[36576] -- Bump Windows Npcap version to v1.76. {issue}36539[36539] {pull}36549[36549] *Packetbeat* @@ -364,6 +301,9 @@ is collected by it. + + + diff --git a/libbeat/docs/release.asciidoc b/libbeat/docs/release.asciidoc index a1bb2ada147..1677789f2ea 100644 --- a/libbeat/docs/release.asciidoc +++ b/libbeat/docs/release.asciidoc @@ -8,6 +8,7 @@ This section summarizes the changes in each release. Also read <> for more detail about changes that affect upgrade. +* <> * <> * <> * <>