initial commit for adding Microsoft Threat Protection package

[P1llus]

What does this PR do?

This PR adds a package for a new module related to Microsoft Threat Protection that can be followed here: elastic/beats#21446

Related issues

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Branch indexing

• Start Time: 2020-12-04T22:28:29.726+0000

• Duration: 21 min 45 sec

Test stats 🧪

Test	Results
Failed	0
Passed	61
Skipped	0
Total	61

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[mtojek]

Let's convert it to draft to prevent from accidental merge.

[mtojek]

I checked out your PR and found out some issue. I fixed them with the mage check command (mtojek@b7d58f0).

Regarding docs, please use the template file: https://github.com/elastic/integrations/blob/master/dev/import-beats-resources/microsoft/docs/README.md . This file is a template for the package's README.

You can either use the testing/environment stack or try the relatively fresh tool to work with integrations: https://github.com/elastic/elastic-package . Once you install it according to the README instruction, simply navigate to the packages/microsoft and run:

elastic-package build
elastic-package stack up -d

^ this command will bring up the Elastic stack with modified Microsoft package and spawn the Elastic Agent (all as docker images).

Please be aware that we need to wait for the new Kibana snapshot image, which will support data stream and policy templates (last renaming operation). I hope it's a matter day as snapshots are built on daily basis.

(you will see available data streams below)

To update stack images you can use:

elastic-package stack update -v

If you have any questions about integrations and elastic-package tool, feel free to mention @ycombinator or me.

[P1llus]

Thanks for your comments @mtojek !

For the documentation, the Microsoft Package already existed, but the module had some new datasets added to it, so when using ImportBeats it overrides certain parts of the documentation, however I restored the format of the documentation back to its original form, so it's a bit weird that the documentation which was already in master was wrong.

No worries though, learning experience for me! Will go through your steps and documentation later today to apply the fixes 👍

[mtojek]

so when using ImportBeats it overrides certain parts of the documentation

Yes, ideally the ImportBeats target should be used only once to import all the stuff you need. Otherwise there is a risk to override/lose your changes.

Please be aware that the elastic-package (used internally via mage) is still experimental, so it may not be perfect. We appreciate any feedback on this one.

[mtojek]

You have two same Github keys defined.

[P1llus]

Yeah just fixing it up, but docker builds are not coming up which im trying to fix first. With both elastic-package and the "old" way.

mage check:

2020/10/19 10:33:48 Notice: the template file "dev/import-beats-resources/log/docs/README.md" does not exist. The README.md file will not be rendered.
>> mod - updating vendor directory
elastic-package stack up -d
Boot up the Elastic stack
Custom build packages directory found: /home/marius/go/src/github.com/elastic/integrations/build/integrations
Error: booting up the stack failed: running docker-compose failed: running command failed: running Docker Compose up command failed: exit status 1
Error: running elastic-package failed: running "/home/marius/go/src/github.com/elastic/integrations/build/elastic-package stack up -d" failed with exit code 

[mtojek]

Try to run:

rm -rf ~/.elastic-package/
elastic-package stack update -v

It's always good to verify it with the CI, which should always use latest Docker images.

[P1llus]

By the way, if a mage check fails on docker, it will not kill the running containers, so you will end up with something like this:

docker ps
CONTAINER ID        IMAGE                                                           COMMAND                  CREATED             STATUS                    PORTS                                NAMES
5c7617fb2eed        elastic-package-stack_package-registry                          "./package-registry …"   13 minutes ago      Up 12 minutes (healthy)   127.0.0.1:8080->8080/tcp             elastic-package-stack_package-registry_1
954a961924da        docker.elastic.co/elasticsearch/elasticsearch:7.10.0-SNAPSHOT   "/tini -- /usr/local…"   13 minutes ago      Up 12 minutes (healthy)   127.0.0.1:9200->9200/tcp, 9300/tcp   elastic-package-stack_elasticsearch_1

It's a bit tricky since the next time you try to run mage check it will fail since the ports are already in use.

[P1llus]

Try to run:

rm -rf ~/.elastic-package/
elastic-package stack update -v

It's always good to verify it with the CI, which should always use latest Docker images.

This resolved my docker issues, will try to push a new complete commit soon once tested.

[mtojek]

@P1llus let us know when the PR is ready for review (currentyl it's marked as draft).

[P1llus]

@mtojek will let you know! Need to go through it with someone from the security ingest team later today first, too many small things to currently cover on comments and I want to ensure it follows the same convention we use for our other security packages.
That way I can apply that to all the packages I have in the pipeline :)

[mtojek]

As there were incompatible changes pushed to the master branch (PR: #410), please rebase this PR against the master branch.

Long story short: dev/import-beats-resources/microsoft/docs/README.md is packages/microsoft/_dev/build/docs/README.md now.