Ingest Pipeline tester using Simulate API #6261

kvch · 2018-02-01T19:05:04Z

This script is tests a single log line or multiple lines against an Ingest pipeline using the Simulate API. It creates a request and displays the response from Elasticsearch.

Usage

Usage of /tmp/go-build407861954/b001/exe/main:
  -elasticsearch string
        Elasticsearch URL (default "http://localhost:9200")
  -log string
        Single log line to test
  -logfile string
        Path to log file
  -maxbytes int
        Number of max bytes to be read (default 10485760)
  -modules string
        Path to modules (default "./modules")
  -multiline.mode string
        Multiline mode (default "before")
  -multiline.negate
        Multiline negate
  -multiline.pattern string
        Multiline pattern
  -pipeline string
        Path to pipeline
  -simulate.verbose
        Call Simulate API with verbose option
  -strict.perms
        Strict permission checking on config files (default true)
  -verbose
        Print full output of Simulate API

Finding pipelines

specify path to one pipeline
specify a directory to look for pipelines
specify a module/fileset pair and find its pipeline

Examples

Correct pipeline

Input pipeline: PostgreSQL/log
Input log: 2017-04-03 22:32:14.322 CEST [31225] postgres@mydb LOG: could not receive data from client: Connection reset by peer

$ go run main.go -pipeline "../../module/postgresql/log/ingest/pipeline.json" -log "2017-04-03 22:32:14.322 CEST [31225] postgres@mydb LOG:  could not receive data from client: Connection reset by peer" -verbose
{
  "docs": [
    {
      "doc": {
        "_id": "id",
        "_index": "index",
        "_ingest": {
          "timestamp": "2018-02-01T18:54:00.450Z"
        },
        "_source": {
          "@timestamp": "2017-04-03T22:32:14.322Z",
          "message": "2017-04-03 22:32:14.322 CEST [31225] postgres@mydb LOG:  could not receive data from client: Connection reset by peer",
          "postgresql": {
            "log": {
              "database": "mydb",
              "level": "LOG",
              "message": "could not receive data from client: Connection reset by peer",
              "thread_id": "31225",
              "timestamp": "2017-04-03 22:32:14.322",
              "timezone": "CEST",
              "user": "postgres"
            }
          }
        },
        "_type": "doc"
      }
    }
  ]
}

Incorrect pipeline

Input pipeline: PostgreSQL/log
Input log: 2017-04-03 22:32:14.322 [31225] postgres@mydb LOG: could not receive data from cl ient: Connection reset by peer

> $ go run main.go -path "../../module/postgresql/log/ingest/pipeline.json" \-log "2017-04-03 22:32:14.322 [31225] postgres@mydb LOG:  could not receive data f
rom client: Connection reset by peer"
{
  "docs": [
    {
      "doc": {
        "_id": "id",
        "_index": "index",
        "_ingest": {
          "timestamp": "2018-02-01T18:56:47.338Z"
        },
        "_source": {
          "error": {
            "message": "Provided Grok expressions do not match field value: [2017-04-03 22:32:14.322 [31225] postgres@mydb LOG:  could not receive data from cl
ient: Connection reset by peer]"
          },
          "message": "2017-04-03 22:32:14.322 [31225] postgres@mydb LOG:  could not receive data from client: Connection reset by peer"
        },
        "_type": "doc"
      }
    }
  ]
}

TODO

Questions

What do you think about the interface?
Do you need more features?

ruflin

This is really cool as it should make testing a pipeline much easier. Having this code in Golang means it could probably also replace parts of our slow "system module tests".

For the naming of the variables / options I suggest to keep them identical to what we have in filebeat. So for multiline it is for example -multiline.match.

The PR reminded me also of #5028 from @urso

ruflin · 2018-02-03T03:58:31Z

filebeat/scripts/tester/main.go

+func getMultiline(s *bufio.Scanner, line string, multiNegate bool, regex *regexp.Regexp) []string {
+	matches := regex.MatchString(line)
+	fullLine := line
+	if matches || !matches && multiNegate {


Could we reuse here the logic of our multiline reader to make sure we have the exact same behaviour?

urso · 2018-02-05T15:05:36Z

💯

The difference between this and #5028 is, I wanted to run tests based on a user module/prospector configuration. Right from config file. This PR requires you to name the pipeline.json file under test. Both have slightly different use-cases, user vs. module developer. Still this command can still be used by users for very custom pipeline configurations.

A mix of both would be nice. E.g.:

if input is pipeline json file -> use it
I pass a directory, lookup the json file:
- if multiple files exist, check all and output which ones fail and which ones succeed
If I pass a modules name only -> resolved module directory and treat like directory

As one can push multiple events via SimulateAPI at once, it might be somewhat daunting to read/interpret the output. It's too easy to miss an error. Plus, for debugging purposes it's helpful to add the verbose flag to the API call (in order get collect per processor input/output). E.g. how about: - only show errored events by default (only failed documents, not all)

have verbose flag to show all contents
have double-verbose flag to set SimulateAPI verbose flag as wel

urso · 2018-02-05T15:06:33Z

filebeat/scripts/tester/main.go

+	r = reader.NewStripNewline(r)
+
+	if multiPattern != "" {
+		p := match.MustCompile(multiPattern)


this panics on invalid pattern. Let's create a somewhat user-friendly error message.

urso · 2018-02-05T15:07:03Z

filebeat/scripts/tester/main.go

+			return nil, err
+		}
+	}
+	r = reader.NewLimit(r, 10485760)


The limit should be configurable. Think, users with very big xml documents :/

urso · 2018-02-12T17:14:52Z

filebeat/scripts/tester/main.go

+		}
+	}
+}
+func getLogsFromFile(logfile, multiPattern string, multiNegate bool, matchMode string, maxBytes int) ([]string, error) {


Kinda scary considering users potentially using a 1GB test file :)

urso · 2018-02-12T17:16:54Z

filebeat/scripts/tester/main.go

+	}
+	defer f.Close()
+
+	encFactory, ok := encoding.FindEncoding("utf8")


encoding should be configurable. Especially windows users will oftentimes have utf16

kvch · 2018-02-13T18:17:46Z

@urso I addressed your review notes.
The CI jobs fail due to ES problems.

urso · 2018-02-13T18:59:40Z

filebeat/scripts/tester/main.go

+
+	encFactory, ok := encoding.FindEncoding(conf.encoding)
+	if !ok {
+		return nil, fmt.Errorf("unable to find 'utf8' encoding")


error message should not say utf8

kvch added in progress Pull request is currently in progress. discuss Issue needs further discussion. review Filebeat Filebeat labels Feb 1, 2018

kvch force-pushed the filebeat/feature/pipeline-tester branch from 633aa23 to 4effa77 Compare February 2, 2018 09:01

ruflin reviewed Feb 3, 2018

View reviewed changes

urso reviewed Feb 5, 2018

View reviewed changes

kvch force-pushed the filebeat/feature/pipeline-tester branch from 651c301 to ece6d39 Compare February 9, 2018 12:48

kvch removed discuss Issue needs further discussion. in progress Pull request is currently in progress. labels Feb 9, 2018

urso reviewed Feb 12, 2018

View reviewed changes

kvch force-pushed the filebeat/feature/pipeline-tester branch from 4f8464f to c12449e Compare February 13, 2018 15:44

urso reviewed Feb 13, 2018

View reviewed changes

urso approved these changes Feb 15, 2018

View reviewed changes

kvch added 12 commits February 19, 2018 13:24

init pipeline tester

aa037ce

add multiline support

8c92622

print error to stderr

45b7e71

rename path option to pipeline

79a7632

fix multiline reading

e6f6742

init using multiline reader of beats

f41581e

change interface to be similar to the config file

1a0f202

address review notes

a01f5b6

use Compile to avoid panics

b33e561

add tests for pipeline finding && minor error handling

715b948

fix goimports

2fe15f9

make encoding configurable && add conf struct

a0b3791

kvch added 2 commits February 19, 2018 13:24

fix encoding error message

6384ded

add @timestamp to test read_timestamp

79ddb1a

kvch force-pushed the filebeat/feature/pipeline-tester branch from 3343785 to 79ddb1a Compare February 19, 2018 12:25

tsg merged commit 1d7e325 into elastic:master Feb 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ingest Pipeline tester using Simulate API #6261

Ingest Pipeline tester using Simulate API #6261

kvch commented Feb 1, 2018 •

edited

Loading

ruflin left a comment

ruflin Feb 3, 2018

kvch Feb 5, 2018

urso commented Feb 5, 2018

urso Feb 5, 2018

urso Feb 5, 2018 •

edited

Loading

urso Feb 12, 2018

urso Feb 12, 2018

kvch commented Feb 13, 2018

urso Feb 13, 2018

Ingest Pipeline tester using Simulate API #6261

Ingest Pipeline tester using Simulate API #6261

Conversation

kvch commented Feb 1, 2018 • edited Loading

Usage

Finding pipelines

Examples

Correct pipeline

Incorrect pipeline

TODO

Questions

ruflin left a comment

Choose a reason for hiding this comment

ruflin Feb 3, 2018

Choose a reason for hiding this comment

kvch Feb 5, 2018

Choose a reason for hiding this comment

urso commented Feb 5, 2018

urso Feb 5, 2018

Choose a reason for hiding this comment

urso Feb 5, 2018 • edited Loading

Choose a reason for hiding this comment

urso Feb 12, 2018

Choose a reason for hiding this comment

urso Feb 12, 2018

Choose a reason for hiding this comment

kvch commented Feb 13, 2018

urso Feb 13, 2018

Choose a reason for hiding this comment

kvch commented Feb 1, 2018 •

edited

Loading

urso Feb 5, 2018 •

edited

Loading