Code

[filebeat] Add a Kafka input#12850

Code

Merged

faec merged 53 commits intoelastic:masterelastic/beats:masterfrom faec:kafka-inputfaec/beats:kafka-inputCopy head branch name to clipboardAug 15, 2019

+1,070-1

Commits

Commits on Jun 26, 2019

• Add kafka input to filebeat import list

  faec

  committed
  c0dbd6a

Commits on Jul 8, 2019

• Initial skeleton of filebeat kafka input

  faec

  committed
  878a9e7

Commits on Jul 10, 2019

• Cleanup

  faec

  committed
  5f6c0d9

• Turn on Wait() and Stop()

  faec

  committed
  1c83d55

• add InitialOffset configuration parameter

  faec

  committed
  7298e9c

• Document new kafka output fields

  faec

  committed
  7345c18

Commits on Jul 11, 2019

• Add username / password and ssl config

  faec

  committed
  b925014

• Add parameter for client id

  faec

  committed
  1f6ecc3

• Add metric registry to sarama reader

  faec

  committed
  207bebe

• Log kafka errors

  faec

  committed
  ec386a0

Commits on Jul 16, 2019

• Add remaining kafka metadata fields

  faec

  committed
  0aaa710

• document new metadata fields

  faec

  committed
  95e6bff

• Adjust kafka producer version / test message in tests

  faec

  committed
  5bb711c

• Don't record BlockTimestamp if it's zero

  faec

  committed
  9601e0c

• Remove debug printf

  faec

  committed
  2d8d37c

Commits on Jul 17, 2019

• make fmt

  faec

  committed
  e6b8b53

Commits on Jul 18, 2019

• Merge branch 'master' into kafka-input

  faec

  committed
  a843dbc

• Add kafka container to filebeat integration tests

  faec

  committed
  426c98f

• regenerate docs

  faec

  committed
  43f1cde

• Remove unused test helpers

  faec

  committed
  da3eb99

• Add header verification to kafka integration test

  faec

  committed
  3da5f99

Commits on Jul 22, 2019

• Addressing review comments

  faec

  committed
  6bdb13f

Commits on Jul 23, 2019

• Review comments

  faec

  committed
  bfeaeb0

• Add several more kafka configuration settings

  faec

  committed
  8061d86

Commits on Jul 25, 2019

• Document kafka input configuration

  faec

  committed
  678d71b

Commits on Jul 26, 2019

• Merge branch 'master' into kafka-input

  faec

  committed
  b8b3446

• Update for new outlet api

  faec

  committed
  6a45e05

Commits on Jul 29, 2019

• Add end-to-end ACK to the kafka input / synchronize access to the session

  faec

  committed
  5cee082

• Update integration test

  faec

  committed
  a16e862

• make integration test an integration test againk not unit

  faec

  committed
  e2137cb

• Replace sarama context with a minimal wrapper suggested by @urso

  faec

  committed
  cd8a3d9

Commits on Jul 31, 2019

• Clarify docs

  faec

  committed
  e54b9d8

• Addressing review comments

  faec

  committed
  418364a

• Fix kafka input Stop()

  faec

  committed
  73aa0bc

Commits on Aug 1, 2019

• revised run loop in progress

  faec

  committed
  76f7792

• refactor groupHandler / fix end-to-end ACK

  faec

  committed
  1a29d7b

• Use strings for kafka headers

  faec

  committed
  c903177

• Make kafka message keys strings on indexing

  faec

  committed
  f5dd360

• Adjust config parameter names

  faec

  committed
  4e73ac8

• Update changed config fields in docs

  faec

  committed
  f2441d8

• Merge branch 'master' into kafka-input

  faec

  committed
  6aa839e

• Add IsolationLevel config option

  faec

  committed
  6ad363e

• Document IsolationLevel

  faec

  committed
  df98c90

Commits on Aug 8, 2019

• working on corrected index template

  faec

  committed
  a90d7c5

Commits on Aug 9, 2019

• add compromise data layout for kafka headers

  faec

  committed
  c302125

• addressing review comments

  faec

  committed
  6ae3210

• use exponential backoff for connecting

  faec

  committed
  43b5bec

• shutdown outlet via the CloseRef config field

  faec

  committed
  62488ef

Commits on Aug 12, 2019

• Fixing backoff wait call

  faec

  committed
  60f436c

• Add wait_close parameter

  faec

  committed
  e223a64

• Update header handling in integration test

  faec

  committed
  135da31

Commits on Aug 15, 2019

• Fix backoff handling again...

  faec

  committed
  5f64da9

• Adjust what the connection backoff responds to

  faec

  committed
  419ddab