Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/master' into k8s-keystore-backend
Browse files Browse the repository at this point in the history
  • Loading branch information
ChrsMark committed May 6, 2020
2 parents d0f3629 + 36d136c commit a0d1181
Show file tree
Hide file tree
Showing 200 changed files with 29,793 additions and 1,618 deletions.
13 changes: 13 additions & 0 deletions .ci/scripts/install-kind.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -exuo pipefail

MSG="parameter missing."
DEFAULT_HOME="/usr/local"
KIND_VERSION=${KIND_VERSION:?$MSG}
HOME=${HOME:?$DEFAULT_HOME}
KIND_CMD="${HOME}/bin/kind"

mkdir -p "${HOME}/bin"

curl -sSLo "${KIND_CMD}" "https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64"
chmod +x "${KIND_CMD}"
14 changes: 14 additions & 0 deletions .ci/scripts/install-kubectl.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -exuo pipefail

MSG="parameter missing."
DEFAULT_HOME="/usr/local"
K8S_VERSION=${K8S_VERSION:?$MSG}
HOME=${HOME:?$DEFAULT_HOME}
KUBECTL_CMD="${HOME}/bin/kubectl"

mkdir -p "${HOME}/bin"

curl -sSLo "${KUBECTL_CMD}" "https://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/linux/amd64/kubectl"
chmod +x "${KUBECTL_CMD}"

13 changes: 0 additions & 13 deletions .ci/scripts/kind-setup.sh
Original file line number Diff line number Diff line change
@@ -1,18 +1,5 @@
#!/usr/bin/env bash
set -exuo pipefail

MSG="parameter missing."
K8S_VERSION=${K8S_VERSION:?$MSG}
HOME=${HOME:?$MSG}
KBC_CMD="${HOME}/bin/kubectl"

mkdir -p "${HOME}/bin"

curl -sSLo "${KBC_CMD}" "https://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/linux/amd64/kubectl"
chmod +x "${KBC_CMD}"

GO111MODULE="on" go get sigs.k8s.io/[email protected]
kind create cluster --image kindest/node:${K8S_VERSION}

export KUBECONFIG="$(kind get kubeconfig-path)"
kubectl cluster-info
8 changes: 7 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,13 @@ jobs:
stage: test
- os: linux
before_install: .ci/scripts/travis_has_changes.sh metricbeat libbeat || travis_terminate 0
env: TARGETS="-C metricbeat integration-tests"
install:
- .ci/scripts/install-kind.sh
- .ci/scripts/install-kubectl.sh
env:
- TARGETS="-C metricbeat integration-tests"
- K8S_VERSION=v1.17.2
- KIND_VERSION=v0.7.0
go: $TRAVIS_GO_VERSION
stage: test
- os: linux
Expand Down
7 changes: 6 additions & 1 deletion CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d

*Filebeat*
- Improve ECS field mappings in panw module. event.outcome now only contains success/failure per ECS specification. {issue}16025[16025] {pull}17910[17910]
- Improve ECS categorization field mappings for nginx module. http.request.referrer is now lowercase & http.request.referrer only populated when nginx sets a value {issue}16174[16174] {pull}17844[17844]
- Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . {issue}16180[16180] {pull}17982[17982]

*Heartbeat*

Expand Down Expand Up @@ -115,6 +117,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. {pull}17425[17425]
- Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. {issue}17734[17734] {pull}17735[17735]
- Fixed `cloudfoundry.access` to have the correct `cloudfoundry.app.id` contents. {pull}17847[17847]
- Fixing `ingress_controller.` fields to be of type keyword instead of text. {issue}17834[17834]
- Fixed typo in log message. {pull}17897[17897]
- Fix Cisco ASA ASA 3020** and 106023 messages {pull}17964[17964]

Expand Down Expand Up @@ -207,6 +210,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add keystore support for autodiscover static configurations. {pull]16306[16306]
- Add Kerberos support to Elasticsearch output. {pull}17927[17927]
- Add k8s keystore backend. {pull}18096[18096]
- Add support for fixed length extraction in `dissect` processor. {pull}17191[17191]

*Auditbeat*

Expand Down Expand Up @@ -276,7 +280,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Enhance `elasticsearch/slowlog` fileset to handle ECS-compatible logs emitted by Elasticsearch. {issue}17715[17715] {pull}17729[17729]
- Improve ECS categorization field mappings in misp module. {issue}16026[16026] {pull}17344[17344]
- Added Unix stream socket support as an input source and a syslog input source. {pull}17492[17492]
- Improve ECS categorization field mappings for nginx module. {issue}16174[16174] {pull}17844[17844]
- Improve ECS categorization field mappings in postgresql module. {issue}16177[16177] {pull}17914[17914]
- Improve ECS categorization field mappings in rabbitmq module. {issue}16178[16178] {pull}17916[17916]
- Improve ECS categorization field mappings in redis module. {issue}16179[16179] {pull}17918[17918]
Expand Down Expand Up @@ -354,9 +357,11 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Reference kubernetes manifests mount data directory from the host when running metricbeat as daemonset, so data persist between executions in the same node. {pull}17429[17429]
- Add more detailed error messages, system tests and small refactoring to the service metricset in windows. {pull}17725[17725]
- Stack Monitoring modules now auto-configure required metricsets when `xpack.enabled: true` is set. {issue}16471[[16471] {pull}17609[17609]
- Add dashboard for the azure database account metricset. {pull}17901[17901]
- Allow partial region and zone name in googlecloud module config. {pull}17913[17913]
- Add aggregation aligner as a config parameter for googlecloud stackdriver metricset. {issue}17141[[17141] {pull}17719[17719]
- Move the perfmon metricset to GA. {issue}16608[16608] {pull}17879[17879]
- Add static mapping for metricsets under aws module. {pull}17614[17614] {pull}17650[17650]

*Packetbeat*

Expand Down
106 changes: 85 additions & 21 deletions Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

@Library('apm@current') _

import groovy.transform.Field

/**
This is required to store the stashed id with the test results to be digested with runbld
*/
@Field def stashedTestReports = [:]

pipeline {
agent { label 'ubuntu && immutable' }
environment {
Expand All @@ -11,6 +18,7 @@ pipeline {
PIPELINE_LOG_LEVEL = "INFO"
DOCKERELASTIC_SECRET = 'secret/observability-team/ci/docker-registry/prod'
DOCKER_REGISTRY = 'docker.elastic.co'
RUNBLD_DISABLE_NOTIFICATIONS = 'true'
}
options {
timeout(time: 2, unit: 'HOURS')
Expand Down Expand Up @@ -39,7 +47,7 @@ pipeline {
options { skipDefaultCheckout() }
steps {
deleteDir()
gitCheckout(basedir: "${BASE_DIR}")
gitCheckout(basedir: "${BASE_DIR}", githubNotifyFirstTimeContributor: true)
stash allowEmpty: true, name: 'source', useDefaultExcludes: false
dir("${BASE_DIR}"){
loadConfigEnvVars()
Expand Down Expand Up @@ -539,14 +547,20 @@ pipeline {
stages {
stage('Generators Metricbeat Linux'){
steps {
makeTarget("Generators Metricbeat Linux", "-C generator/_templates/metricbeat test")
makeTarget("Generators Metricbeat Linux", "-C generator/_templates/metricbeat test-package")
// FIXME see https://github.com/elastic/beats/issues/18132
catchError(buildResult: 'SUCCESS', message: 'Ignore error temporally', stageResult: 'UNSTABLE') {
makeTarget("Generators Metricbeat Linux", "-C generator/_templates/metricbeat test")
makeTarget("Generators Metricbeat Linux", "-C generator/_templates/metricbeat test-package")
}
}
}
stage('Generators Beat Linux'){
steps {
makeTarget("Generators Beat Linux", "-C generator/_templates/beat test")
makeTarget("Generators Beat Linux", "-C generator/_templates/beat test-package")
// FIXME see https://github.com/elastic/beats/issues/18132
catchError(buildResult: 'SUCCESS', message: 'Ignore error temporally', stageResult: 'UNSTABLE') {
makeTarget("Generators Beat Linux", "-C generator/_templates/beat test")
makeTarget("Generators Beat Linux", "-C generator/_templates/beat test-package")
}
}
}
stage('Generators Metricbeat Mac OS X'){
Expand All @@ -559,7 +573,10 @@ pipeline {
}
}
steps {
makeTarget("Generators Metricbeat Mac OS X", "-C generator/_templates/metricbeat test")
// FIXME see https://github.com/elastic/beats/issues/18132
catchError(buildResult: 'SUCCESS', message: 'Ignore error temporally', stageResult: 'UNSTABLE') {
makeTarget("Generators Metricbeat Mac OS X", "-C generator/_templates/metricbeat test")
}
}
}
stage('Generators Beat Mac OS X'){
Expand All @@ -572,7 +589,10 @@ pipeline {
}
}
steps {
makeTarget("Generators Beat Mac OS X", "-C generator/_templates/beat test")
// FIXME see https://github.com/elastic/beats/issues/18132
catchError(buildResult: 'SUCCESS', message: 'Ignore error temporally', stageResult: 'UNSTABLE') {
makeTarget("Generators Beat Mac OS X", "-C generator/_templates/beat test")
}
}
}
}
Expand All @@ -587,13 +607,16 @@ pipeline {
}
}
steps {
k8sTest(["v1.16.2","v1.15.3","v1.14.6","v1.13.10","v1.12.10","v1.11.10"])
k8sTest(["v1.18.2","v1.17.2","v1.16.4","v1.15.7","v1.14.10"])
}
}
}
}
}
post {
always {
runbld()
}
cleanup {
notifyBuildResult(prComment: true)
}
Expand Down Expand Up @@ -669,9 +692,7 @@ def withBeatsEnv(boolean archive, Closure body) {
dockerLogin(secret: "${DOCKERELASTIC_SECRET}", registry: "${DOCKER_REGISTRY}")
}
dir("${env.BASE_DIR}") {
sh(label: "Install Go ${GO_VERSION}", script: ".ci/scripts/install-go.sh")
sh(label: "Install docker-compose ${DOCKER_COMPOSE_VERSION}", script: ".ci/scripts/install-docker-compose.sh")
sh(label: "Install Mage", script: "make mage")
installTools()
// TODO (2020-04-07): This is a work-around to fix the Beat generator tests.
// See https://github.com/elastic/beats/issues/17787.
setGitConfig()
Expand All @@ -682,7 +703,7 @@ def withBeatsEnv(boolean archive, Closure body) {
} finally {
if (archive) {
catchError(buildResult: 'SUCCESS', stageResult: 'UNSTABLE') {
junit(allowEmptyResults: true, keepLongStdio: true, testResults: "**/build/TEST*.xml")
junitAndStore(allowEmptyResults: true, keepLongStdio: true, testResults: "**/build/TEST*.xml")
archiveArtifacts(allowEmptyArchive: true, artifacts: '**/build/TEST*.out')
}
}
Expand All @@ -709,21 +730,32 @@ def withBeatsEnvWin(Closure body) {
deleteDir()
unstash 'source'
dir("${env.BASE_DIR}"){
bat(label: "Install Go/Mage/Python ${GO_VERSION}", script: ".ci/scripts/install-tools.bat")
installTools()
try {
if(!params.dry_run){
body()
}
} finally {
catchError(buildResult: 'SUCCESS', stageResult: 'UNSTABLE') {
junit(allowEmptyResults: true, keepLongStdio: true, testResults: "**\\build\\TEST*.xml")
junitAndStore(allowEmptyResults: true, keepLongStdio: true, testResults: "**\\build\\TEST*.xml")
archiveArtifacts(allowEmptyArchive: true, artifacts: '**\\build\\TEST*.out')
}
}
}
}
}

def installTools() {
def i = 2 // Number of retries
if(isUnix()) {
retry(i) { sh(label: "Install Go ${GO_VERSION}", script: ".ci/scripts/install-go.sh") }
retry(i) { sh(label: "Install docker-compose ${DOCKER_COMPOSE_VERSION}", script: ".ci/scripts/install-docker-compose.sh") }
retry(i) { sh(label: "Install Mage", script: "make mage") }
} else {
retry(i) { bat(label: "Install Go/Mage/Python ${GO_VERSION}", script: ".ci/scripts/install-tools.bat") }
}
}

def goos(){
def labels = env.NODE_LABELS

Expand Down Expand Up @@ -806,14 +838,14 @@ def dumpFilteredEnvironment(){
def k8sTest(versions){
versions.each{ v ->
stage("k8s ${v}"){
withEnv(["K8S_VERSION=${v}"]){
withEnv(["K8S_VERSION=${v}", "KIND_VERSION=v0.7.0", "KUBECONFIG=${env.WORKSPACE}/kubecfg"]){
withGithubNotify(context: "K8s ${v}") {
withBeatsEnv(false) {
sh(label: "Install k8s", script: """
eval "\$(gvm use ${GO_VERSION} --format=bash)"
.ci/scripts/kind-setup.sh
""")
sh(label: "Kubernetes Kind",script: "make KUBECONFIG=\"\$(kind get kubeconfig-path)\" -C deploy/kubernetes test")
sh(label: "Install kind", script: ".ci/scripts/install-kind.sh")
sh(label: "Install kubectl", script: ".ci/scripts/install-kubectl.sh")
sh(label: "Integration tests", script: "MODULE=kubernetes make -C metricbeat integration-tests")
sh(label: "Setup kind", script: ".ci/scripts/kind-setup.sh")
sh(label: "Deploy to kubernetes",script: "make -C deploy/kubernetes test")
sh(label: 'Delete cluster', script: 'kind delete cluster')
}
}
Expand Down Expand Up @@ -880,7 +912,7 @@ def loadConfigEnvVars(){
env.GO_VERSION = readFile(".go-version").trim()

withEnv(["HOME=${env.WORKSPACE}"]) {
sh(label: "Install Go ${env.GO_VERSION}", script: ".ci/scripts/install-go.sh")
retry(2) { sh(label: "Install Go ${env.GO_VERSION}", script: ".ci/scripts/install-go.sh") }
}

// Libbeat is the core framework of Beats. It has no additional dependencies
Expand Down Expand Up @@ -980,3 +1012,35 @@ def setGitConfig(){
def isDockerInstalled(){
return sh(label: 'check for Docker', script: 'command -v docker', returnStatus: true)
}

def junitAndStore(Map params = [:]){
junit(params)
// STAGE_NAME env variable could be null in some cases, so let's use the currentmilliseconds
def stageName = env.STAGE_NAME ? env.STAGE_NAME.replaceAll("[\\W]|_",'-') : "uncategorized-${new java.util.Date().getTime()}"
stash(includes: params.testResults, allowEmpty: true, name: stageName, useDefaultExcludes: true)
stashedTestReports[stageName] = stageName
}

def runbld() {
catchError(buildResult: 'SUCCESS', message: 'runbld post build action failed.') {
if (stashedTestReports) {
dir("${env.BASE_DIR}") {
sh(label: 'Prepare workspace context',
script: 'find . -type f -name "TEST*.xml" -path "*/build/*" -delete')
// Unstash the test reports
stashedTestReports.each { k, v ->
dir(k) {
unstash v
}
}
sh(label: 'Process JUnit reports with runbld',
script: '''\
cat >./runbld-script <<EOF
echo "Processing JUnit reports with runbld..."
EOF
/usr/local/bin/runbld ./runbld-script
'''.stripIndent()) // stripIdent() requires '''/
}
}
}
}
9 changes: 9 additions & 0 deletions NOTICE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1371,6 +1371,15 @@ License type (autodetected): Apache-2.0
Apache License 2.0


--------------------------------------------------------------------
Dependency: github.com/docker/spdystream
Revision: 449fdfce4d96
License type (autodetected): Apache-2.0
./vendor/github.com/docker/spdystream/LICENSE:
--------------------------------------------------------------------
Apache License 2.0


--------------------------------------------------------------------
Dependency: github.com/dop251/goja
Overwrite: github.com/andrewkroh/goja
Expand Down
14 changes: 13 additions & 1 deletion dev-tools/mage/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,19 @@ func dockerInfo() (*DockerInfo, error) {
// PATH.
func HaveDockerCompose() error {
_, err := exec.LookPath("docker-compose")
return errors.Wrap(err, "docker-compose was not found on the PATH")
if err != nil {
return fmt.Errorf("docker-compose is not available")
}
return nil
}

// HaveKubectl returns an error if kind is not found on the PATH.
func HaveKubectl() error {
_, err := exec.LookPath("kubectl")
if err != nil {
return fmt.Errorf("kubectl is not available")
}
return nil
}

// FindReplace reads a file, performs a find/replace operation, then writes the
Expand Down
Loading

0 comments on commit a0d1181

Please sign in to comment.