Cherrypick #10176 to 6.x branch: [Filebeat] Iptables / ubiquiti module #10461
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Remove release state override * Remove override for release state in Journalbeat
…astic#9067) (elastic#9073) (cherry picked from commit 936949d)
) (cherry picked from commit 7c1e6b6)
…ing to fetch stats (elastic#9095) Cherry-pick of PR elastic#9003 to 6.x branch. Original message: Resolves elastic#8915. This PR teaches the `Fetch()` method of the `elasticsearch/ccr` metricset to first check if the CCR feature is available per the current Elasticsearch license. If it isn't, the metricset logs+reports an **actionable** error message **every minute**. Before this PR, there was no such check so the call to the CCR stats API would simply fail with a 403 error from Elasticsearch if Elasticsearch wasn't using a Trial or Platinum license. That unhelpful 403 error would get logged+reported every 10s (or whatever `period` the metricset was configured to). ## Before this PR ### In the `metricbeat` logs: <img width="839" alt="screen shot 2018-11-08 at 4 42 28 pm" src="https://user-images.githubusercontent.com/51061/48236039-5fa30c80-e375-11e8-8d9d-64c4861fb35d.png"> ### In the `metricbeat-*` index: <img width="1491" alt="screen shot 2018-11-08 at 4 42 38 pm" src="https://user-images.githubusercontent.com/51061/48236036-5b76ef00-e375-11e8-9a21-7cde7620c80b.png"> ## After this PR ### In the `metricbeat` logs: <img width="1236" alt="screen shot 2018-11-08 at 4 35 19 pm" src="https://user-images.githubusercontent.com/51061/48235932-e4415b00-e374-11e8-8b39-a60e4fe87279.png"> ### In the `metricbeat-*` index: <img width="1495" alt="screen shot 2018-11-08 at 4 37 27 pm" src="https://user-images.githubusercontent.com/51061/48235909-d095f480-e374-11e8-9eea-718c0deacafa.png">
This adds support for the new TLS version negotiation mechanism introduced in TLS 1.3. It relies on a new extension: `supported_versions`. When this extension is used in the CLIENT_HELLO message, it features a list of versions the client is willing to use: ``` "supported_versions": [ "TLS 1.3", "TLS 1.2", "TLS 1.1", "TLS 1.0" ], ``` If the server supports the extension, it will pick one of the offered versions and include it in the SERVER_HELLO message: ``` "supported_versions": "TLS 1.3", ``` The TLS parser will report a new field, `tls.version`, that is the TLS version that has been selected after negotiation, either using the new negotiation introduced in TLS 1.3 or the legacy negotiation mechanism that used the version field in HELLO messages. Updated the TLS dashboard to use the new version field: - Server version visualization changed to TLS Version - Client version is not useful anymore, replaced by tls.server_certificate.public_key_size Fixes elastic#8647 (cherry picked from commit 51c1aa2)
…same files definition (elastic#9104) Cherry-pick of PR elastic#8836 to 6.x branch. Original message: This PR make sure the definition style used by the packages.yml and the custom function packages.yml match.
* Add statement about required license * Add fixes from review * Put license statemetn in shared file * Add license statemetn * Update libbeat/docs/shared-license-statement.asciidoc Co-Authored-By: dedemorton <[email protected]>
(cherry picked from commit c791a59)
) (elastic#9065) This will also ensure that credentials are installed before healthcheck goes healthy. Fixes elastic#7769 (cherry picked from commit a8230bf)
`panelJSON` is/can be an array, so marshaling to MapStr caused problems. I think this was the root cause for elastic#8952. (cherry picked from commit 5546e8b)
… elasticsearch/index_recovery metricset (elastic#9175) Cherry-pick of PR elastic#9164 to 6.x branch. Original message: This PR teaches the `elasticsearch/index_recovery` metricset to index the Elasticsearch `cluster_uuid` and `cluster_name` as the module-level `cluster.id` and `cluster.name` fields, respectively.
The download link in the Kafka Docker image was not valid anymore. (cherry picked from commit 36f76a3)
Through updating requests to 2.20.0 some of the python dependencies have become incompatible. This was introduced in elastic#8808. Not sure if it actually caused issues but warnings were shown when running make update. (cherry picked from commit 6fb6f2a)
… elasticsearch/pending_tasks metricset (elastic#9174) Cherry-pick of PR elastic#9166 to 6.x branch. Original message: This PR teaches the `elasticsearch/pending_tasks` metricset to index the Elasticsearch `cluster_uuid` and `cluster_name` as the module-level `cluster.id` and `cluster.name` fields, respectively.
… elasticsearch/ml_job metricset (elastic#9173) * Index cluster.id and cluster.name in elasticsearch/ml_job metricset (elastic#9165) This PR teaches the `elasticsearch/ml_job` metricset to index the Elasticsearch `cluster_uuid` and `cluster_name` as the module-level `cluster.id` and `cluster.name` fields, respectively. (cherry picked from commit e7f4a1c) * Fixing field name for 6.x
…ation tests (elastic#9197) Cherry-pick of PR elastic#9183 to 6.x branch. Original message: This PR takes the separate `logstash/node/node_integration_test.go` and `logstash/node_stats/node_stats_integration_test.go` files and consolidates them into a single `logstash/logstash_integration_test.go` file, in the same vein as [`elasticsearch/elasticsearch_integration_test.go`](https://github.com/elastic/beats/blob/master/metricbeat/module/elasticsearch/elasticsearch_integration_test.go). For good measure, this PR also runs the consolidated integration test and re-generates `data.json` files for the `node` and `node_stats` metricsets.
Fix panic on docker healthcheck collection on dockers without healthchecks. (cherry picked from commit 1c43d6c)
…feFileRotate (elastic#9141) * Propagate Sync error when running SafeFileRotate (elastic#9069) Previously, it was possible SafeFileRotate encountered an error without propagating it, because the return value of os.Sync was not utilized. From now on the errors from Sync are propagated. (cherry picked from commit 93eb92a)
Restore broken panelsJSON from the PR elastic#8675. (cherry picked from commit 4d718fa)
…lastic#9070) This PR adds back the missing `metricset.namespace` field in to payloads generated by http server. (cherry picked from commit 08dfd03) Co-authored-by: Vijay Samuel <[email protected]>
…9098) Cherry-pick of PR elastic#9062 to 6.x branch. Original message: We are now using Mockbeat and the file output instead of using the ES output and we now assert the Keystore log message, this should make the test more stable and faster to run. Fix the following: ``` test_keystore_with_present_key failed 3 times: elastic+beats+master+multijob-windows/beat=libbeat,label=windows failed 2 times elastic+beats+6.x+multijob-darwin/beat=libbeat,label=macosx failed 1 times test_keystore_with_nested_key failed 2 times: elastic+beats+6.5+multijob-windows/beat=libbeat,label=windows failed 1 times elastic+beats+master+multijob-windows/beat=libbeat,label=windows failed 1 times ```
…cs collection per core (elastic#9218) (cherry picked from commit 52df22d)
… Bytes from the io.Reader (elastic#9236) * Adding a Debug reader to detect Null Bytes from the io.Reader (elastic#9210) When you are reading logs on a network volume like NFS or ZFS it is possible that the underlying filesystem return null bytes instead of returning concrete data, its not currently possible to detect that in all scenario unless events are eventually send to ES and you can inspect them and see \u0000 chars in the messages. This is a small proposal to add a Debug Reader which should only by used for debugging purpose it allow to log if any null bytes are present in the streams of bytes and will log surround values. It accepts an io.Reader as the source of data, a buffer size, a predicate to check the value of a byte and how much detection invokation it should do before disabling the check. Enable it with either of the following selectors: "*" or "detect_null_bytes" (cherry picked from commit 96c924a)
See elastic#7690 (cherry picked from commit be187b9)
…astic#9206) This PR teaches the `elasticsearch/node` metricset to index the Elasticsearch `cluster_uuid` as the module-level `cluster.id` field. (cherry picked from commit 9bd2499)
… with ES ccr_stats API changes (elastic#9201) * Updating elasticsearch/ccr metricset with ES ccr_stats API changes (elastic#9179) Resolves elastic/kibana#25778. The ES CCR API changed a little before the 6.5.0 FF and we didn't catch these changes 😞. This PR updates the CCR metricset code to work with the new ES CCR API response structure. ## Testing this PR 1. Setup CCR: https://www.elastic.co/guide/en/elastic-stack-overview/current/ccr-getting-started.html 2. Checkout this PR and `make` Metricbeat: `cd metricbeat; make` 3. Enable the `elasticsearch` module: `./metricbeat modules enable elasticsearch` 4. Edit `modules.d/elasticsearch.yml` and make sure the `ccr` metricset is enabled and `xpack.enabled: true` is set. 5. Start Metricbeat: `./metricbeat -e` 6. After about 10-20 seconds, check that Metricbeat indexed the correct document of `type` = `ccr_stats` into `.monitoring-es-mb-6-*`. * Renaming field for 6.x
(cherry picked from commit 238e55d)
…tic#10326) (cherry picked from commit 0cbfefd)
* Fixing event type field to be non-ECS * Fix parsing of URI field
The `fields` was missing for the x-pack/libbeat build target. (cherry picked from commit 3f49f61)
…astic#10179) (elastic#10259) * Enable back CM integration suite This reverts commit edeed09. And do the following: - Move all docker-compose.yml file to the version 2.3 format to have support for `start_period` - The health check check the cluster health instead of checking that the host respond. - Use the `ELASTIC_PASSWORD` variable instead of invoking the CLI. - DUMP the last health check information and the docker-compose logs
* Amending pipeline to handle structured logs * Adding globs for structured audit log files * Fixing up pipeline * Fixing up log fixture * Only build URI field if it's not going to be empty * Fixing up log fixture * Updating fields.ymls * Splitting test logs and expected files * Fixing up test fixtures * Reverting content in original expected test file * Fixing offsets after splitting files * Updating generated files * Fixing up pipeline * Regenerating golden files * Using multiple pipelines * Regenerating golden files * Adding CHANGELOG entries * Regenerating generated files * Updating golden files for 6.x * Removing file accidentally ported over from master * Rebasing...
…ent metricsets as GA (elastic#10343) * Release kubernetes apiserver and event metricsets as GA (elastic#10212) Rest of the module was GA already (cherry picked from commit f95ec5d) * Update CHANGELOG.next.asciidoc
…10388) Partial backport of #elastic#9777. This PR ncreases the dynamic script compilations for Filebeat system tests to `1000/1m`.
Follow up to elastic#10352 per elastic#10352 (comment): > While working on this PR I realized that we don't have sample lines for the **structured** elasticsearch audit log containing a request body (which is supposed to be parsed into the `http.request.body.content` field). I'm working with `@albertzaharovits` to get such a sample and will incorporate it into follow up PRs (for `master` and `6.x`). Accordingly, this PR adds sample lines to the structured and unstructured log file test fixtures for the `elasticsearch/audit` fileset and teaches the fileset to parse any new fields encountered in these sample lines.
…elastic#10344) This commit add support to publish a function that listen to a Kinesis stream using the CLI. Features: - Kinesis can now be defined in the YML. - AWS installer can now define a custom policies to be added to the lambda role. - Kinesis support `TRIM_HORIZON` and `LATEST` as the starting position strategy, `AT_TIMESTAMP` is currently not supported because the cloudformation API doesn't accept a TIMESTAMP when configuring the subscription. - Kinesis allow to configure the batch size. (cherry picked from commit 5b4bb7f)
Cherry-picks community beat additions into the 6.x branch.
… container. (elastic#10380) (elastic#10409) As a retrospect leaving the invocation of `docker-compose logs` was a bad idea, first it could hide some error in the integration suite and it could add unnecessary long logs line in the CI job making debugging even harder. So instead, I've remove it completely and leave up to the person debugging to enabled it back. (cherry picked from commit fd149bb)
**NOTES:** This PR is based on top of elastic#10116 Correctly add the permissions to the lambda role when monitoring SQS queue. Fixes: elastic#9152 (cherry picked from commit dc963c4)
…lastic#10361) Bugfix: Include System module in configuration files on Windows. (cherry picked from commit e8013bd)
) (elastic#10331) [Heartbeat] Incorporate factory metadata for autodiscover Heartbeat factories get metadata from autodiscover and other sources. This change automatically adds that data to events keeping heartbeat behavior in-line with other beats. (cherry picked from commit 8f4e186)
elastic#10394) Fixes elastic#9415 by closing the db handle. (cherry picked from commit f06cf46) Co-authored-by: Jaime A <[email protected]>
This adds a new x-pack module to parse iptables logs from a file.
The module understands additional fields added by Ubiquiti firewalls.
Example iptables event:
```
"iptables": {
"ether_type": 2048,
"input_device": "eth0",
"output_device": "wan1",
"tcp": {
"reserved_bits": 0,
"flags": "ACK",
"window": 128
},
"precedence_bits": 0,
"length": 123,
"tos": 0,
"output_device": "lo",
"id": 12345,
"ttl": 255
},
```
For Ubiquiti logs, it parses the Ubiquiti tag before iptables logs.
Format: RuleSet-RuleNumber-Action
It's a common pattern to format the RuleSet as "InputDevice-OutputDevice".
Action can be A for allow and D for Deny. RuleNumber can be a number
or "default".
For [source-dest-42-D]:
```
"iptables.ubiquiti.input_zone": "source",
"iptables.ubiquiti.output_zone": "dest",
"iptables.ubiquiti.rule_number": "42",
"iptables.ubiquiti.rule_set": "source-dest",
"event.outcome": "deny",
```
The following ECS fields are populated:
```
"destination.ip": "10.4.0.5",
"destination.mac": "90:10:20:76:8d:20",
"destination.port": 443,
"ecs.version": "1.0.0-beta2",
"event.dataset": "iptables.log",
"event.module": "iptables",
"event.outcome": "deny", (only when Ubiquiti extension found)
"fileset.name": "log",
"network.transport": "tcp",
"network.type": "ipv4",
"source.geo.city_name": "Bellaterra",
"source.geo.continent_name": "Europe",
"source.geo.country_iso_code": "ES",
"source.geo.location.lat": 41.5026,
"source.geo.location.lon": 2.0875,
"source.geo.region_iso_code": "ES-B",
"source.geo.region_name": "Barcelona",
"source.ip": "158.109.0.1",
"source.mac": "90:10:65:29:b6:2a",
"source.port": 38842
```
Closes elastic#8781
Co-Authored-by: James Spiteri <[email protected]>
(cherry picked from commit d7c14df)
houndci-bot
reviewed
Jan 31, 2019
| func (fs *Fileset) GetPipeline(esVersion string) (pipelineID string, content map[string]interface{}, err error) { | ||
| path, err := applyTemplate(fs.vars, fs.manifest.IngestPipeline, false) | ||
| // GetPipelines returns the JSON content of the Ingest Node pipeline that parses the logs. | ||
| func (fs *Fileset) GetPipelines(esVersion common.Version) (pipelines []pipeline, err error) { |
There was a problem hiding this comment.
exported method GetPipelines returns unexported type []fileset.pipeline, which can be annoying to use
|
Wrong branch |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #10176 to 6.x branch. Original message:
This adds a new x-pack module to parse iptables logs from a file. The module understands additional fields added by Ubiquiti firewalls.
Example iptables event:
For Ubiquiti logs, it parses the Ubiquiti tag before iptables logs.
Format: RuleSet-RuleNumber-Action
It's a common pattern to format the RuleSet as "InputZone-OutputZone". Action can be A for allow and D for Deny. RuleNumber can be a number or "default".
For [source-dest-42-D]:
The following ECS fields are populated: