Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #18802 to 7.7: Add missing network.sent_packets_count metric into compute googlecloud #18833

Closed
wants to merge 889 commits into from
Closed

Conversation

kaiyan-sheng
Copy link
Contributor

Cherry-pick of PR #18802 to 7.x branch. Original message:

This PR is to add 4 memory metrics(only for e2 family VMs) into googlecloud compute metricset and also added the missing network.sent_packets_count metric.

jsoriano and others added 30 commits March 12, 2020 20:16
…16846)

When dealing with autodiscover & hints, especially, that generate new configurations depending on the discovered workloads, having the `layouts` as part of the debug message is quite useful for checking what the timestamp processor is trying to match against the target field.

(cherry picked from commit 9ff5bc3)

Co-authored-by: Jorge Luis Betancourt Gonzalez <[email protected]>
- event.kind
- event.type

Closes #16164

(cherry picked from commit b82a427)
#16959)

- event.kind
- convert pipeline to yaml

Closes #16163

(cherry picked from commit a91cc9e)
… (#16106)

* Add CustomString mapping to CEF for Forcepoint NGFW

Closes #14663

(cherry picked from commit 82b8fe6)
)

Add dashboards to the Winlogbeat Security module for visualizing User Management events
and Group Management events. There are two versions of each dashboard - one with and without TSVB (time series visual builder) visualizations.

This updates the Winlogbeat build to include the dashboards from the module directories.

Add it adds winlog.event_data.MemberName to the fields.yml because it's used in the user
management dashboard and should be in the fields.yml so the dashboards load without error.

Co-authored-by: Andrew Kroh <[email protected]>
(cherry picked from commit d866824)

Co-authored-by: Anabella Cristaldi <[email protected]>
* Fix docs for translate_sid processor (#16961)

I forgot to change the conditional for ifdef to ifndef. It was also missing a changelog entry.

(cherry picked from commit 4a082e7)
…g copied. (#16987) (#17002)

* Fix issue where autodiscover hints default configuration was not being copied.

* Add changelog.

* Add test and update comment.

(cherry picked from commit 661ff14)
- related.hash
- related.ip
- related.user
- fix description

Closes #16157
Closes #16289

(cherry picked from commit 3e6edf2)
* major refactor to support different systemd version

* format and updates

* update ref docs

* update ref, again

* add newline

* Fix error string

Co-Authored-By: Jaime Soriano Pastor <[email protected]>

* add changelog entry

* make update

* add build target

Co-authored-by: Jaime Soriano Pastor <[email protected]>
(cherry picked from commit 1d36da7)
#16986)

- event.kind
- event.category
- event.type
- event.outcome
- lowercase http.request.method
- host.id
- host.name
- related.user

Closes #16160

(cherry picked from commit 3c13de5)
(cherry picked from commit 0d45df1)

Co-authored-by: Edu González de la Herrán <[email protected]>
This adds metadata to the Windows .exe files that the build produces. This makes our binaries a little more friendly to automation on Windows because you can check the version programmatically with ease. It also makes is easy for end user to inspect some details about the file from the file properties dialog in Windows.

Closes #15232

For example:

```
PS C:\vagrant\winlogbeat > (Get-Info .\winlogbeat.exe).VersionInfo | Format-List

OriginalFilename  : winlogbeat.exe
FileDescription   : Winlogbeat ships Windows event logs to Elasticsearch or Logstash.
ProductName       : Winlogbeat
Comments          : commit=8d6cf58f347579188d707421da6b70b2f66701ea
CompanyName       : Elastic
FileName          : C:\vagrant\winlogbeat\winlogbeat.exe
FileVersion       : 8.0.0
ProductVersion    : 8.0.0
IsDebug           : False
IsPatched         : False
IsPreRelease      : False
IsPrivateBuild    : False
IsSpecialBuild    : False
Language          : Language Neutral
LegalCopyright    : Copyright Elastic, License ASL 2.0
LegalTrademarks   :
PrivateBuild      :
SpecialBuild      :
FileVersionRaw    : 8.0.0.0
ProductVersionRaw : 8.0.0.0
```

(cherry picked from commit 695b167)
* Update vendored elastic/ecs to v1.5.0

* Update fields.ecs.yml to v1.5.0

* Fix encoding issue in fields' docs

Fields' docs generation is failing in some systems when non-ascii
characters appear in the fields descriptions and/or examples.

* Patch fields.ecs.yml to avoid parser trouble

The fields docs generator was having trouble representing an example as
an object in a way that was consistent across different python3 versions
due to keys in a dict being serialised to string with different
orderings. This caused checks to fail.

(cherry picked from commit 76dcd34)
Tests on stats API can finish even before the mocked beat has completed
its initialization, so it can receive the signal to stop before it can
handle signals, terminating with an status code -15 (killed by SIGTERM),
instead of the expected status code 0.

Check that the beat has been completely started before trying to kill
it, and move common code to setUp/tearDown.

Fix also an incorrect error message.

(cherry picked from commit 825cf05)
…6990)

* Improve ECS field mappings in suricata module

- destination.domain
- dns.question.top_level_domain
- event.category
- event.kind
- event.outcome
- event.type
- related.hash
- related.ip
- rule.category
- rule.id
- rule.name
- tls.client.server_name
- tls.resumed
- tls.server.certificate
- tls.server.certificate_chain
- tls.server.hash.sha1
- tls.server.issuer
- tls.server.ja3s
- tls.server.not_after
- tls.server.not_before
- tls.server.subject
- tls.version
- tls.version_protocol

Closes #16181

(cherry picked from commit 7eb2fba)
…t publisher and reader (#16912)

* Remove global logger from libbeat publisher and reader (#16886)


(cherry picked from commit ad2672d)

* backport #16915 in this PR
…odule (#16978)

* [Filebeat] Add cloudwatch fileset in aws module (#16579)

* Add cloudwatch fileset in aws module

(cherry picked from commit ad4597c)
Add build tags to AWS and Azure integration tests, and
make tests fail if they cannot get a working configuration.

Add support for a `TEST_TAGS` environment variable in
mage targets that add additional build tags to go tests calls.
When `TEST_TAGS` includes a cloud provider, it is expected
that valid credentials to run these tests are available in the
environment, if not, tests will fail. This is intentional, to avoid
skipping tests by mistake due to some misconfiguration in CI.

(cherry picked from commit 1345a97)
This PR adds the option to remove files from the vendor folder during `mage update`.

`yacc` is triggered from time to time when `mage vendor` copies files in a particular order. It leads to build failures unexpectedly.
(cherry picked from commit 9370128)
…16866) (#16893)

Switch from Debug to Error when unrecoveral events happen and
add extra debug messages when indexing and matching pods.

(cherry picked from commit 1d6323f)

Co-authored-by: Flavio Percoco <[email protected]>
We should not be using a sum on kubernetes.pod.network.rx.bytes,
kubernetes.pod.network.tx.bytes, and kubernetes.container.cpu.usage.core.ns;
these fields are counters.


(cherry picked from commit 0a2b3f4)
…17051)

* Release ActiveMQ filebeat/metricbeat module as GA

(cherry picked from commit bb81b23)
* Add missing logger init in New() functions

(cherry picked from commit e64a18f)
mikemadden42 and others added 21 commits May 13, 2020 07:35
The shared pipeline for Cisco ASA and FTD creates temporary fields under
the _temp_ object. If a failure happens in the middle of the pipeline,
all those fields would be indexed, causing the index mapping to grow too
big.

Fixes #18391

(cherry picked from commit 3605c47)
 for googlecloud audit log (#18554)

* Add additional fields to address issue #18465 for googlecloud audit log (#18472)
…18546)

Sometimes the mapped source/destination IP field of an event is a hostname
instead of an IP address. This caused ingestion of the event to fail.

This patch makes the asa-ftd-pipeline to only populate those fields when
a valid IP address is found.

In the future we may want to revisit this if .nat.domain or
.nat.address fields become available.

(cherry picked from commit b24ed97)
…es (#18525) (#18533)

PANW pipeline was mapping:

Bytes sent to client/destination bytes.
Bytes received to server/source bytes.
Packets sent to server/destination packets.
Packets received to client/source packets.

All of these mappings are wrong.

The correct is:
Bytes sent to client/source bytes.
Bytes received to server/destination bytes.
Packets sent to client/source packets.
Packets received to server/destination packets.

Also adding some missing ECS mappings in the process.

Fixes #18522

(cherry picked from commit 463a52a)
… ingress_controller (#18569)

* Fix source.address not being set for nginx ingress_controller

Signed-off-by: chrismark <[email protected]>
Co-authored-by: chendo <[email protected]>
(cherry picked from commit 8e29ab7)
…h metricset in aws module (#18618)

* [Metricbeat] Fix tags_filter for cloudwatch metricset in aws module (#18524)

* Fix tags_filter for cloudwatch metricset
* if tags_filter is given, overwrite tags in cloudwatch specific config

(cherry picked from commit 5aa3d0c)
…kets (#18603)

The O365 Management API is generating some events that contain a
`ClientIP` / `ClientIPAddress` field consisting of an IPv4 address
enclosed in square brackets. This is breaking ingestion of those events
as the brackets are only stripped for IPv6 addresses.

> "ClientIP": "[10.11.12.13]:12345"

This patch adds support for IPv4 enclosed in brackets and a few other
edge cases.

Fixes #18587

(cherry picked from commit eaf196d)
This addresses a problem with crypto/ssh/terminal used as secure input
under Windows.

Fixes #18649

(cherry picked from commit 2c90b6a)
…18735)

* Change index_pattern from "" to "metricbeat-*"

(cherry picked from commit 1d62f17)
* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes #18154

(cherry picked from commit 87c3ad3)
@kaiyan-sheng kaiyan-sheng requested review from a team as code owners May 28, 2020 21:06
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label Team:Automation Label for the Observability productivity team and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 28, 2020
@kaiyan-sheng kaiyan-sheng deleted the backport_18802_7.7 branch May 28, 2020 21:06
@kaiyan-sheng kaiyan-sheng restored the backport_18802_7.7 branch May 28, 2020 21:08
@kaiyan-sheng kaiyan-sheng deleted the backport_18802_7.7 branch May 29, 2020 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Automation Label for the Observability productivity team
Projects
None yet
Development

Successfully merging this pull request may close these issues.