Simplify Running Test in Beats

[ruflin]

Currently beats has a testsuite written in Golang and one in Python. Both test suites can be run with an environment to include integration tests or not. Unfortunately the commands are not very consistent. To simplify the testing I suggest we reduce it to three commands and all the other things are dependent on environment variables set or not:

• make tests: Runs all White box tests (Golang)
• make system-tests: Runs all system tests (Python)
• make testsuite: Runs both

To decide which tests should be run, environment variables should be used:

• INTEGRATION: Run tests which require external services
• RACE: Enable race detection
• ENVIRONMENT: Run tests inside an docker environment
• COVERAGE: Enable coverage report
• BENCHMARK: Benchmarkt tests
• LOAD: Load tests

The nice property of having these as environment variables is that each beat can overwrite the defaults and can decide if make tests should by default run integration tests or not. Also it allows easily to add new ENV variables which are specific to beats.

Examples

Some examples of before and after.

• make system-tests-environment -> INTEGRATION=1 ENVIONMENT=1 COVERAGE=1 make system-tests
• make integration-tests-environment -> INTEGRATION=1 ENVIONMENT=1 COVERAGE=1 make tests
• make unit-tests -> COVERAGE=1 make tests

Change in behavior

Currently when running make integration-tests in beats the non integration tests are not executed because of the // +build !integration build flag in the tests. But I think we should change this behaviour to also run them when INTEGRATION env variable is enabled. This simplifies the implementation.

[exekias]

Sounds good to me, I like the use of env vars to switch behaviors, and unify Makefile.

Something I would add as a goal is to ensure tests can be easily run without the Makefile. So if I want to run a specific test, everything needed is managed by the test itself. That means permissions, containers, and so on.

Should help a lot on developing, also for external contributors, as it standardizes testing tools (to nose + go test)

[ruflin]

Agree. I assume that is where #4962 comes into play.

[exekias]

I hope so xD, still work in progress

[ph]

I am going to close this, the test runnings has been completely change.