Merge remote-tracking branch 'upstream/master' into feature/windows-2016

* upstream/master: feat: prepare release pipelines (elastic#21238) Add IP validation to Security module (elastic#21325) Fixes for new 7.10 rsa2elk datasets (elastic#21240) o365input: Restart after fatal error (elastic#21258) Fix panic in cgroups monitoring (elastic#21355) Handle multiple upstreams in ingress-controller (elastic#21215) [CI] Fix runbld when workspace does not exist (elastic#21350) [Filebeat] Fix checkpoint (elastic#21344) [CI] Archive build reasons (elastic#21347) Add dashboard for pubsub metricset in googlecloud module (elastic#21326) [Elastic Agent] Allow embedding of certificate (elastic#21179) Adds a default for failure_cache.min_ttl (elastic#21085) [libbeat] Disk queue implementation (elastic#21176)
v1v · Sep 29, 2020 · 6696ff1 · 6696ff1
2 parents 835e53a + 0cbf4e5
commit 6696ff1
Show file tree

Hide file tree

Showing 53 changed files with 13,184 additions and 14,487 deletions.
diff --git a/.ci/jobs/beats-release-minor-major.yml b/.ci/jobs/beats-release-minor-major.yml
@@ -0,0 +1,20 @@
+---
+- job:
+    name: Beats/Release/beats-release-minor-major
+    display-name: 'Prepare Major/minor Release'
+    description: 'Automate the steps to prepare a new Release branch'
+    view: Beats
+    project-type: pipeline
+    pipeline-scm:
+        script-path: release_scripts/pipeline-release-minor-major.groovy
+        scm:
+        - git:
+            url: [email protected]:elastic/ingest-dev.git
+            refspec: +refs/heads/*:refs/remotes/origin/* +refs/pull/*/head:refs/remotes/origin/pr/*
+            wipe-workspace: 'True'
+            name: origin
+            shallow-clone: true
+            credentials-id: f6c7695a-671e-4f4f-a331-acdce44ff9ba
+            reference-repo: /var/lib/jenkins/.git-references/ingest-dev.git
+            branches:
+                - master
diff --git a/.ci/jobs/beats-release-patch.yml b/.ci/jobs/beats-release-patch.yml
@@ -0,0 +1,20 @@
+---
+- job:
+    name: Beats/Release/beats-release-patch
+    display-name: 'Prepare Patch Release'
+    description: 'Automate the steps to prepare a new Patch'
+    view: Beats
+    project-type: pipeline
+    pipeline-scm:
+        script-path: release_scripts/pipeline-release-patch.groovy
+        scm:
+        - git:
+            url: [email protected]:elastic/ingest-dev.git
+            refspec: +refs/heads/*:refs/remotes/origin/* +refs/pull/*/head:refs/remotes/origin/pr/*
+            wipe-workspace: 'True'
+            name: origin
+            shallow-clone: true
+            credentials-id: f6c7695a-671e-4f4f-a331-acdce44ff9ba
+            reference-repo: /var/lib/jenkins/.git-references/ingest-dev.git
+            branches:
+                - master
diff --git a/.ci/jobs/folders.yml b/.ci/jobs/folders.yml
@@ -4,3 +4,8 @@
     name: Beats
     description: Beats
     project-type: folder
+
+- job:
+    name: Beats/Release
+    description: Jobs for release preparation
+    project-type: folder
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -22,6 +22,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Added `certificate` TLS verification mode to ignore server name mismatch. {issue}12283[12283] {pull}20293[20293]
 - Autodiscover doesn't generate any configuration when a variable is missing. Previously it generated an incomplete configuration. {pull}20898[20898]
 - Remove redundant `cloudfoundry.*.timestamp` fields. This value is set in `@timestamp`. {pull}21175[21175]
+- Allow embedding of CAs, Certificate of private keys for anything that support TLS in ouputs and inputs https://github.com/elastic/beats/pull/21179
 
 *Auditbeat*
 
@@ -174,6 +175,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - [Autodiscover] Handle input-not-finished errors in config reload. {pull}20915[20915]
 - Explicitly detect missing variables in autodiscover configuration, log them at the debug level. {issue}20568[20568] {pull}20898[20898]
 - Fix `libbeat.output.write.bytes` and `libbeat.output.read.bytes` metrics of the Elasticsearch output. {issue}20752[20752] {pull}21197[21197]
+- The `o365input` and `o365` module now recover from an authentication problem or other fatal errors, instead of terminating. {pull}21259[21258]
 
 *Auditbeat*
 
@@ -267,7 +269,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Remove wrongly mapped `tls.client.server_name` from `fortinet/firewall` fileset. {pull}20983[20983]
 - Fix an error updating file size being logged when EOF is reached. {pull}21048[21048]
 - Fix error when processing AWS Cloudtrail Digest logs. {pull}21086[21086] {issue}20943[20943]
+- Handle multiple upstreams in ingress-controller. {pull}21215[21215]
 - Provide backwards compatibility for the `append` processor when Elasticsearch is less than 7.10.0. {pull}21159[21159]
+- Fix checkpoint module when logs contain time field. {pull}20567[20567]
 
 *Heartbeat*
 
@@ -363,6 +367,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 - Fix invalid IP addresses in DNS query results from Sysmon data. {issue}18432[18432] {pull}18436[18436]
 - Fields from Winlogbeat modules were not being included in index templates and patterns. {pull}18983[18983]
+- Add source.ip validation for event ID 4778 in the Security module. {issue}19627[19627]
 
 *Functionbeat*
 
@@ -717,6 +722,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Sanitize `event.host`. {pull}21022[21022]
 - Add overview and platform health dashboards to Cloud Foundry module. {pull}21124[21124]
 - Release lambda metricset in aws module as GA. {issue}21251[21251] {pull}21255[21255]
+- Add dashboard for pubsub metricset in googlecloud module. {pull}21326[21326] {issue}17137[17137]
 
 *Packetbeat*
 

diff --git a/Jenkinsfile b/Jenkinsfile
@@ -107,22 +107,17 @@ pipeline {
                 mapParallelTasks["${k}"] = v
               }
             }
+            notifyBuildReason()
             parallel(mapParallelTasks)
           }
         }
       }
-      post {
-        always {
-          dir("${BASE_DIR}"){
-            // Archive the markdown files that contain the build reasons
-            archiveArtifacts(allowEmptyArchive: false, artifacts: 'build-reasons/*.md')
-          }
-        }
-      }
     }
   }
   post {
     always {
+      deleteDir()
+      unstashV2(name: 'source', bucket: "${JOB_GCS_BUCKET}", credentialsId: "${JOB_GCS_CREDENTIALS}")
       runbld(stashedTestReports: stashedTestReports, project: env.REPO)
     }
     cleanup {
@@ -546,6 +541,17 @@ def isDockerInstalled(){
   }
 }
 
+/**
+* Notify the build reason.
+*/
+def notifyBuildReason() {
+  // Archive the build reason here, since the workspace can be deleted when running the parallel stages.
+  archiveArtifacts(allowEmptyArchive: true, artifacts: 'build-reasons/*.*')
+  if (isPR()) {
+    echo 'TODO: Add a comment with the build reason (this is required to be implemented in the shared library)'
+  }
+}
+
 /**
 * This class is the one used for running the parallel stages, therefore
 * its arguments are passed by the beatsStages step.

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -105135,6 +105135,46 @@ type: array
 
 --
 
+*`nginx.ingress_controller.upstream_address_list`*::
++
+--
+An array of the upstream addresses. It is a list because it is common that several upstream servers were contacted during request processing.
+
+
+type: keyword
+
+--
+
+*`nginx.ingress_controller.upstream.response.length_list`*::
++
+--
+An array of upstream response lengths. It is a list because it is common that several upstream servers were contacted during request processing.
+
+
+type: keyword
+
+--
+
+*`nginx.ingress_controller.upstream.response.time_list`*::
++
+--
+An array of upstream response durations. It is a list because it is common that several upstream servers were contacted during request processing.
+
+
+type: keyword
+
+--
+
+*`nginx.ingress_controller.upstream.response.status_code_list`*::
++
+--
+An array of upstream response status codes. It is a list because it is common that several upstream servers were contacted during request processing.
+
+
+type: keyword
+
+--
+
 *`nginx.ingress_controller.http.request.length`*::
 +
 --
@@ -105182,7 +105222,7 @@ type: keyword
 *`nginx.ingress_controller.upstream.response.length`*::
 +
 --
-The length of the response obtained from the upstream server
+The length of the response obtained from the upstream server. If several servers were contacted during request process, the summary of the multiple response lengths is stored.
 
 
 type: long
@@ -105194,7 +105234,7 @@ format: bytes
 *`nginx.ingress_controller.upstream.response.time`*::
 +
 --
-The time spent on receiving the response from the upstream server as seconds with millisecond resolution
+The time spent on receiving the response from the upstream as seconds with millisecond resolution. If several servers were contacted during request process, the summary of the multiple response times is stored.
 
 
 type: double
@@ -105206,40 +105246,40 @@ format: duration
 *`nginx.ingress_controller.upstream.response.status_code`*::
 +
 --
-The status code of the response obtained from the upstream server
+The status code of the response obtained from the upstream server. If several servers were contacted during request process, only the status code of the response from the last one is stored in this field.
 
 
 type: long
 
 --
 
-*`nginx.ingress_controller.http.request.id`*::
+*`nginx.ingress_controller.upstream.ip`*::
 +
 --
-The randomly generated ID of the request
+The IP address of the upstream server. If several servers were contacted during request process, only the last one is stored in this field.
 
 
-type: keyword
+type: ip
 
 --
 
-*`nginx.ingress_controller.upstream.ip`*::
+*`nginx.ingress_controller.upstream.port`*::
 +
 --
-The IP address of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas.
+The port of the upstream server. If several servers were contacted during request process, only the last one is stored in this field.
 
 
-type: ip
+type: long
 
 --
 
-*`nginx.ingress_controller.upstream.port`*::
+*`nginx.ingress_controller.http.request.id`*::
 +
 --
-The port of the upstream server.
+The randomly generated ID of the request
 
 
-type: long
+type: keyword
 
 --
 

diff --git a/filebeat/module/nginx/fields.go b/filebeat/module/nginx/fields.go
diff --git a/filebeat/module/nginx/ingress_controller/_meta/fields.yml b/filebeat/module/nginx/ingress_controller/_meta/fields.yml
@@ -11,6 +11,26 @@
         Real source IP is restored to `source.ip`.
 
     # ingress-controller specific fields
+    - name: upstream_address_list
+      type: keyword
+      description: >
+        An array of the upstream addresses. It is a list because it is common that several upstream servers
+        were contacted during request processing.
+    - name: upstream.response.length_list
+      type: keyword
+      description: >
+        An array of upstream response lengths. It is a list because it is common that several upstream servers
+        were contacted during request processing.
+    - name: upstream.response.time_list
+      type: keyword
+      description: >
+        An array of upstream response durations. It is a list because it is common that several upstream servers
+        were contacted during request processing.
+    - name: upstream.response.status_code_list
+      type: keyword
+      description: >
+        An array of upstream response status codes. It is a list because it is common that several upstream servers
+        were contacted during request processing.
     - name: http.request.length
       type: long
       format: bytes
@@ -33,28 +53,33 @@
       type: long
       format: bytes
       description: >
-        The length of the response obtained from the upstream server
+        The length of the response obtained from the upstream server. If several servers were contacted during request process,
+        the summary of the multiple response lengths is stored.
     - name: upstream.response.time
       type: double
       format: duration
       description: >
-        The time spent on receiving the response from the upstream server as seconds with millisecond resolution
+        The time spent on receiving the response from the upstream as seconds with millisecond resolution.
+        If several servers were contacted during request process, the summary of the multiple response times is stored.
     - name: upstream.response.status_code
       type: long
       description: >
-        The status code of the response obtained from the upstream server
-    - name: http.request.id
-      type: keyword
-      description: >
-        The randomly generated ID of the request
+        The status code of the response obtained from the upstream server. If several servers were contacted during
+        request process, only the status code of the response from the last one is stored in this field.
     - name: upstream.ip
       type: ip
       description: >
-        The IP address of the upstream server. If several servers were contacted during request processing, their addresses are separated by commas.
+        The IP address of the upstream server. If several servers were contacted during request process,
+        only the last one is stored in this field.
     - name: upstream.port
       type: long
       description: >
-        The port of the upstream server.
+        The port of the upstream server. If several servers were contacted during request process,
+        only the last one is stored in this field.
+    - name: http.request.id
+      type: keyword
+      description: >
+        The randomly generated ID of the request
 
     - name: body_sent.bytes
       type: alias