Skip to content

Commit

Permalink
Merge branch 'feature/use-with-kind-k8s-env' of github.com:v1v/beats …
Browse files Browse the repository at this point in the history
…into feature/use-with-kind-k8s-env

* 'feature/use-with-kind-k8s-env' of github.com:v1v/beats: (52 commits)
  ci: home is declared within withBeatsEnv
  ci: use withKindEnv step
  ci: use getBranchesFromAliases and support next-patch-8 (elastic#30400)
  Update fields.yml (elastic#29609)
  Heartbeat: fix browser metrics and trace mappings (elastic#30258)
  Apply light edits to 8.0 changelog (elastic#30351)
  packetbeat/beater: make sure Npcap installation runs before interfaces are needed (elastic#30396)
  Add a ring-buffer reporter to libbeat (elastic#28750)
  Osquerybeat: Add install verification for osquerybeat (elastic#30388)
  update windows matrix support (elastic#30373)
  Refactor of metricbeat process-gathering metrics and system/process (elastic#30076)
  adjust next changelog wording (elastic#30371)
  [Metricbeat] azure: move event report into loop validDim loop (elastic#29945)
  fix: report GitHub Check before the cache (elastic#30372)
  Add support for non-unique keys in Kafka output headers (elastic#30369)
  ci: 6 major branch reached EOL (elastic#30357)
  reduce Elastic Agent shut down time by stopping processes concurrently (elastic#29650)
  [Filebeat] Add message to register encode/decode debug logs (elastic#30271)
  [libbeat] kafka message header support (elastic#29940)
  Heartbeat: set duration to zero for syntax errors (elastic#30227)
  ...
  • Loading branch information
v1v committed Feb 21, 2022
2 parents 9e73041 + 6ae9e8b commit c20f180
Show file tree
Hide file tree
Showing 254 changed files with 5,711 additions and 4,074 deletions.
2 changes: 1 addition & 1 deletion .ci/jobs/beats.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
discover-pr-forks-strategy: 'merge-current'
discover-pr-forks-trust: 'permission'
discover-pr-origin: 'merge-current'
head-filter-regex: '(main|6\.[89]|7\.1[6789]|8\.\d+|PR-.*|v\d+\.\d+\.\d+)'
head-filter-regex: '(main|7\.1[6789]|8\.\d+|PR-.*|v\d+\.\d+\.\d+)'
discover-tags: true
notification-context: "beats-ci"
repo: 'beats'
Expand Down
28 changes: 2 additions & 26 deletions .ci/schedule-daily.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ pipeline {
stages {
stage('Nighly beats builds') {
steps {
runBuilds(quietPeriodFactor: 2000, branches: ['main', '8.<minor>', '7.<minor>', '7.<next-minor>'])
runBuilds(quietPeriodFactor: 2000, branches: ['main', '8.<minor>', '8.<next-patch>', '7.<minor>'])
}
}
}
Expand All @@ -32,14 +32,7 @@ pipeline {
}

def runBuilds(Map args = [:]) {
def branches = []
// Expand macros and filter duplicated matches.
args.branches.each { branch ->
def branchName = getBranchName(branch)
if (!branches.contains(branchName)) {
branches << branchName
}
}
def branches = getBranchesFromAliases(aliases: args.branches)

def quietPeriod = 0
branches.each { branch ->
Expand All @@ -48,20 +41,3 @@ def runBuilds(Map args = [:]) {
quietPeriod += args.quietPeriodFactor
}
}

def getBranchName(branch) {
// special macro to look for the latest minor version
if (branch.contains('8.<minor>')) {
return bumpUtils.getMajorMinor(bumpUtils.getCurrentMinorReleaseFor8())
}
if (branch.contains('8.<next-minor>')) {
return bumpUtils.getMajorMinor(bumpUtils.getNextMinorReleaseFor8())
}
if (branch.contains('7.<minor>')) {
return bumpUtils.getMajorMinor(bumpUtils.getCurrentMinorReleaseFor7())
}
if (branch.contains('7.<next-minor>')) {
return bumpUtils.getMajorMinor(bumpUtils.getNextMinorReleaseFor7())
}
return branch
}
28 changes: 2 additions & 26 deletions .ci/schedule-weekly.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ pipeline {
stages {
stage('Weekly beats builds') {
steps {
runBuilds(quietPeriodFactor: 1000, branches: ['main', '8.<minor>', '7.<minor>', '7.<next-minor>'])
runBuilds(quietPeriodFactor: 1000, branches: ['main', '8.<minor>', '8.<next-patch>', '7.<minor>'])
}
}
}
Expand All @@ -32,14 +32,7 @@ pipeline {
}

def runBuilds(Map args = [:]) {
def branches = []
// Expand macros and filter duplicated matches.
args.branches.each { branch ->
def branchName = getBranchName(branch)
if (!branches.contains(branchName)) {
branches << branchName
}
}
def branches = getBranchesFromAliases(aliases: args.branches)

def quietPeriod = 0
branches.each { branch ->
Expand All @@ -48,20 +41,3 @@ def runBuilds(Map args = [:]) {
quietPeriod += args.quietPeriodFactor
}
}

def getBranchName(branch) {
// special macro to look for the latest minor version
if (branch.contains('8.<minor>')) {
return bumpUtils.getMajorMinor(bumpUtils.getCurrentMinorReleaseFor8())
}
if (branch.contains('8.<next-minor>')) {
return bumpUtils.getMajorMinor(bumpUtils.getNextMinorReleaseFor8())
}
if (branch.contains('7.<minor>')) {
return bumpUtils.getMajorMinor(bumpUtils.getCurrentMinorReleaseFor7())
}
if (branch.contains('7.<next-minor>')) {
return bumpUtils.getMajorMinor(bumpUtils.getNextMinorReleaseFor7())
}
return branch
}
44 changes: 0 additions & 44 deletions .github/workflows/elastic-agent-pull-requests.yml

This file was deleted.

82 changes: 82 additions & 0 deletions .github/workflows/qa-labels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
name: Add QA labels to Elastic Agent issues
on:
pull_request:
types:
- closed

jobs:
fetch_issues_to_label:
runs-on: ubuntu-latest
# Only run on PRs that were merged for the Elastic Agent teams
if: |
github.event.pull_request.merged_at &&
(
contains(github.event.pull_request.labels.*.name, 'Team:Elastic-Agent') ||
contains(github.event.pull_request.labels.*.name, 'Team:Elastic-Agent-Data-Plane') ||
contains(github.event.pull_request.labels.*.name, 'Team:Elastic-Agent-Control-Plane')
)
outputs:
matrix: ${{ steps.issues_to_label.outputs.value }}
label_ids: ${{ steps.label_ids.outputs.value }}
steps:
- uses: octokit/[email protected]
id: closing_issues
with:
query: |
query closingIssueNumbersQuery($prnumber: Int!) {
repository(owner: "elastic", name: "beats") {
pullRequest(number: $prnumber) {
closingIssuesReferences(first: 10) {
nodes {
id
labels(first: 20) {
nodes {
id
name
}
}
}
}
}
}
}
prnumber: ${{ github.event.number }}
token: ${{ secrets.GITHUB_TOKEN }}
- uses: sergeysova/jq-action@v2
id: issues_to_label
with:
# Map to the issues' node id
cmd: echo $CLOSING_ISSUES | jq -c '.repository.pullRequest.closingIssuesReferences.nodes | map(.id)'
multiline: true
env:
CLOSING_ISSUES: ${{ steps.closing_issues.outputs.data }}
- uses: sergeysova/jq-action@v2
id: label_ids
with:
# Get list of version labels on pull request and map to label's node id, append 'QA:Ready For Testing' id ("LA_kwDOAPya887jWb9x")
cmd: echo $PR_LABELS | jq -c 'map(select(.name | test("v[0-9]+\\.[0-9]+\\.[0-9]+")) | .node_id) + ["LA_kwDOAPya887jWb9x"]'
multiline: true
env:
PR_LABELS: ${{ toJSON(github.event.pull_request.labels) }}

label_issues:
needs: fetch_issues_to_label
runs-on: ubuntu-latest
# For each issue closed by the PR run this job
strategy:
matrix:
issueNodeId: ${{ fromJSON(needs.fetch_issues_to_label.outputs.matrix) }}
name: Label issue ${{ matrix.issueNodeId }}
steps:
- uses: octokit/[email protected]
id: add_labels_to_closed_issue
with:
query: |
mutation add_label($issueid:String!, $labelids:[String!]!) {
addLabelsToLabelable(input: {labelableId: $issueid, labelIds: $labelids}) {
clientMutationId
}
}
issueid: ${{ matrix.issueNodeId }}
labelids: ${{ needs.fetch_issues_to_label.outputs.label_ids }}
token: ${{ secrets.GITHUB_TOKEN }}
9 changes: 9 additions & 0 deletions .mergify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@ queue_rules:
conditions:
- check-success=beats-ci/pr-merge
pull_request_rules:
- name: self-assign PRs
conditions:
- -merged
- -closed
- "#assignee=0"
actions:
assign:
add_users:
- "{{ author }}"
- name: forward-port patches to main branch
conditions:
- merged
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG-developer.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ The list below covers the major changes between 7.0.0-rc2 and main only.
- Add support for `credentials_json` in `gcp` module, all metricsets {pull}29584[29584]
- Add gcp firestore metricset. {pull}29918[29918]
- Added TESTING_FILEBEAT_FILEPATTERN option for filebeat module pytests {pull}30103[30103]
- Add gcp dataproc metricset. {pull}30008[30008]

==== Deprecated

Expand Down
51 changes: 25 additions & 26 deletions CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,47 +12,46 @@ https://github.com/elastic/beats/compare/v7.17.0...v8.0.0[View commits]
*Affecting all Beats*

- Remove the deprecated `xpack.monitoring.*` settings. Going forward only `monitoring.*` settings may be used. {issue}9424[9424] {pull}18608[18608]
- Remove deprecated/undocumented IncludeCreatorMetadata setting from kubernetes metadata config options {pull}28006[28006]
- Remove deprecated fields from kubernetes module {pull}28046[28046]
- Remove deprecated config option aws_partition. {pull}28120[28120]
- Remove deprecated/undocumented `IncludeCreatorMetadata` setting from kubernetes metadata config options. {pull}28006[28006]
- Remove deprecated fields from kubernetes module. {pull}28046[28046]
- Remove deprecated config option `aws_partition`. {pull}28120[28120]
- Improve stats API by adding host metadata. {pull}27963[27963]
- Libbeat: logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. {issue}15544[15544] {pull}28573[28573]
- Remove `auto` from the available options of `setup.ilm.enabled` and set the default value to `true`. {pull}28671[28671]
- add_process_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]
- add_docker_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]
- Use data streams instead of indices for storing events from Beats. {pull}28450[28450]
- Remove option `setup.template.type` and always load composable template with data streams. {pull}28450[28450]
- Remove several ILM options (`rollover_alias` and `pattern`) as data streams does not require index aliases. {pull}28450[28450]
- Index template's default_fields setting is only populated with ECS fields. {pull}28596[28596] {issue}28215[28215]
- Remove `auto` from the available options of `setup.ilm.enabled` and set the default value to `true`. {pull}28671[28671]
- Remove several ILM options (`rollover_alias` and `pattern`) as data streams do not require index aliases. {pull}28450[28450]
- Populate index template's `default_fields` setting with ECS fields only. {pull}28596[28596] {issue}28215[28215]
- Remove deprecated `--template` and `--ilm-policy` flags. Use `--index-management` instead. {pull}28870[28870]
- Remove options `logging.files.suffix` and default to datetime endings in log file names. The format of the new name is `{beatname}-{date}(-n)?.ndjson`. Exmaple log files names from oldest to newest: `filebeat-20200101.ndjson`, `filebeat-20200101-1.ndjson`, `filebeat-20200101-2.ndjson`. {pull}28927[28927]
- Allign kubernetes configuration settings. {pull}29908[29908]
- The extension of the log files of Beats and Elastic Agent is changed to `.ndjson`. If you are collecting the logs, you must change the path configuration to `/path/to/logs/{beatname}*.ndjson` to avoid any issues. {pull}28927[28927]
- Remove `logging.files.suffix` option, and default to datetime endings in log file names. The format of the new name is `{beatname}-{date}(-n)?.ndjson`. Example log file names from oldest to newest: `filebeat-20200101.ndjson`, `filebeat-20200101-1.ndjson`, `filebeat-20200101-2.ndjson`. {pull}28927[28927]
- Align kubernetes configuration settings. {pull}29908[29908]
- Change log file extension for Beats and Elastic Agent to `.ndjson`. If you are collecting the logs, you must change the path configuration to `/path/to/logs/{beatname}*.ndjson` to avoid any issues. {pull}28927[28927]
- Remove legacy support for SSLv3. {pull}30071[30071]

*Filebeat*

- Add `while_pattern` type to multiline reader. {pull}19662[19662]
- auditd dataset: Use process.args to store program arguments instead of auditd.log.aNNN fields. {pull}29601[29601]
- Remove deprecated old awscloudwatch input name. {pull}29844[29844]
- auditd dataset: Use `process.args` to store program arguments instead of `auditd.log.aNNN` fields. {pull}29601[29601]
- Remove deprecated old `awscloudwatch` input name. {pull}29844[29844]

*Metricbeat*

- Remove network and diskio metrics from ec2 metricset. {pull}28316[28316]
- Rename read/write_io.ops_per_sec to read/write.iops in rds metricset. {pull}28350[28350]
- Rename `read/write_io.ops_per_sec` to `read/write.iops` in rds metricset. {pull}28350[28350]
- system/process metricset: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]

*Packetbeat*

- `event.category` no longer contains the value `network_traffic` because this is not a valid ECS event category value. {pull}20556[20556]
- Remove deprecated TLS fields in favor of tls.server.x509 and tls.client.x509 ECS fields. {pull}28487[28487]
- Remove deprecated TLS fields in favor of `tls.server.x509` and `tls.client.x509` ECS fields. {pull}28487[28487]
- HTTP: The field `http.request.method` will maintain its original case. {pull}28620[28620]

*Winlogbeat*

- Remove top level `hash` property from sysmon events {pull}20653[20653]
- Move module processing from local Javascript processor to ingest node {issue}29184[29184] {pull}29435[29435]
- Remove top level `hash` property from sysmon events. {pull}20653[20653]
- Move module processing from local Javascript processor to ingest node. {issue}29184[29184] {pull}29435[29435]

==== Bugfixes

Expand All @@ -62,15 +61,15 @@ https://github.com/elastic/beats/compare/v7.17.0...v8.0.0[View commits]

*Filebeat*

- Fix using log_group_name_prefix in aws-cloudwatch input. {pull}29695[29695]
- Fix using `log_group_name_prefix` in `aws-cloudwatch` input. {pull}29695[29695]

*Heartbeat*

- Add fonts to support more different types of characters for multiple languages. {pull}29861[29861]
- Add fonts to support more types of characters for multiple languages. {pull}29861[29861]

*Metricbeat*

- Extract correct index property in kibana.stats metricset {pull}29622[29622]
- Extract correct index property in `kibana.stats` metricset. {pull}29622[29622]
- Fixed bug with `elasticsearch/cluster_stats` metricset not recording license expiration date correctly. {pull}29711[29711]

*Packetbeat*
Expand All @@ -84,32 +83,32 @@ https://github.com/elastic/beats/compare/v7.17.0...v8.0.0[View commits]

*Affecting all Beats*

- Add config option `rotate_on_startup` to file output {issue}19150[19150] {pull}19347[19347]
- Add config option `rotate_on_startup` to file output. {issue}19150[19150] {pull}19347[19347]
- Update to ECS 8.0 fields. {pull}28620[28620]
- Support custom analyzers in fields.yml. {issue}28540[28540] {pull}28926[28926]
- Support self signed certificates on outputs {pull}29229[29229]
- Support custom analyzers in `fields.yml`. {issue}28540[28540] {pull}28926[28926]
- Support self-signed certificates on outputs. {pull}29229[29229]
- Add FIPS configuration option for all AWS API calls. {pull}[28899]
- Warn users when connecting to older versions of Elasticsearch instances. {pull}29723[29723]
- `add_fields` processor is now able to set metadata in events {pull}30092[30092]
- `add_fields` processor is now able to set metadata in events. {pull}30092[30092]

*Auditbeat*

- system/process: Prevent hashing files in other mnt namespaces. {issue}25777[25777] {issue}29678[29678] {pull}29786[29786]

*Metricbeat*

- Preliminary AIX support {pull}27954[27954]
- Add option to skip older k8s events {pull}29396[29396]
- Add preliminary AIX support. {pull}27954[27954]
- Add option to skip older k8s events. {pull}29396[29396]
- Add `elasticsearch.cluster.id` field to Beat and Kibana modules. {pull}29577[29577]
- Add `elasticsearch.cluster.id` field to Logstash module. {pull}29625[29625]

*Winlogbeat*

- Add support for sysmon event ID 26; FileDeleteDetected. {issue}26280[26280] {pull}29957[29957]
- Add support for sysmon event ID 26; `FileDeleteDetected`. {issue}26280[26280] {pull}29957[29957]

*Elastic Log Driver*

- Fixed docs for hosts {pull}23644[23644]
- Fixed docs for hosts. {pull}23644[23644]

[[release-notes-7.17.0]]
=== Beats version 7.17.0
Expand Down
Loading

0 comments on commit c20f180

Please sign in to comment.