[Beats Management] APIs

[ycombinator]

Each Beat should be able to enroll itself with Kibana before users can start centrally managing that beat's configuration from within Kibana. For this purpose, Beats will need a set of enrollment Kibana APIs they can call.

Here is what the enrollment sequence involving a Beat user, a Beat, Kibana, and an administrative user will look like:

Once enrolled, a user (with appropriate security privileges) should be able to sign-in to the Kibana UI and centrally-manage enrolled Beats and their configurations.

The following HTTP APIs should exist in order to enable the aforementioned enrollment and central management operations:

Method + Resource	Purpose	Security required	Notes	PR
POST /api/beats/enrollment_tokens	Create new enrollment token(s)	beats_admin role	* Should write to Kibana audit log file	#19018 (merged)
POST /api/beats/agent/{beat UUID}	Enroll Beat (in unverified state)	Enrollment token (shared secret)	* Should check if specified enrollment token matches a stored one * Should delete stored enrollment token so it may not be reused * Response should include access token * Should write to Kibana audit log file	#19056 (merged)
GET /api/beats/agents	List enrolled Beats (verified or not)	beats_admin role		#19086 (merged)
POST /api/beats/agents/verify	Verify enrolled Beat(s)	beats_admin role	* Should mark Beat as verified * Should write to Kibana audit log file	#19103 (merged)
DELETE /api/beats/agents	Disenroll Beat(s)	beats_admin role	* Should delete stored Beat public key and mark Beat as unverified * Should keep all Beat's tag associations around * Should write to Kibana audit log file
PUT /api/beats/agent/{beat UUID}	Update an enrolled Beat's information	Access token	* Should write to Kibana audit log file	#19148 (merged)
PUT /api/beats/tag/{tag}	Create or update a tag, containing 0 or more configuration blocks	beats_admin role	* Should write to Kibana audit log file	#19342 (merged)
POST /api/beats/agents_tags/assignments	Assign tag(s) to Beat(s)	beats_admin role	* Should write to Kibana audit log file	#19431 (merged)
POST /api/beats/agents_tags/removals	Remove tag(s) from Beat(s)	beats_admin role	* Should write to Kibana audit log file	#19440 (merged)
GET /api/beats/tags	List tags	beats_admin role
GET /api/beats/tag/{tag}	Retrieve tag information, including all the configuration blocks for it	beats_admin role
DELETE /api/beats/tags	Delete tag(s), including all associated configuration blocks	beats_admin role	* Should write to Kibana audit log file
GET /api/beats/agent/{beat UUID}/configuration	Retrieve merged configuration for the given Beat	Access token	* Should use caching + E-Tag/If-None-Match	#20603
GET /api/beats/agent/{beat UUID}	Retrieve a single Beat's information	beats_admin role

Persistence

The APIs will use an internal Elasticsearch indices for persistence. These indices should come into existence via an index template that is loaded up before any of the above APIs become available for use. PR #19072 (merged).

Security

Also, we should provide a built-in X-Pack Security role, beats_admin, that will provide total access to persisted data. elastic/elasticsearch#30520.

[exekias]

I think permissions in PUT /enrollment/agent/{beat UUID} (admin permissions required) are different, admin permission is not required, but a valid token.

If we are going for short-lived one-time tokens we don't need verification, wdyt?

[ycombinator]

Yes, you're right @exekias. Fixed. Thanks!

[ycombinator]

@exekias I've expanded the scope of this issue beyond just Enrollment APIs to include Management APIs, persistence, and security since these concerns are all related. Specifically, I've updated the issue description with the following:

• I expanded the scope of this issue to cover enrollment and management APIs both, just to make it easier for me to have one place to get a quick summary of all APIs we need to create in Kibana and make sure we have all our needs covered.

• I expanded the CRUD APIs on beats and tags into specific, individual APIs, again just to get a more complete picture to make sure I don't miss implementing something.

• I added a section on Persistence to show how I imagine the information will be stored in Elasticsearch and how it will be accessed (i.e Security).

• I added a section on Security detailing what X-Pack Security roles I think we should provide out of the box for convenience. I also tied the roles mentioned here to each API (see "Security required" column in the API table).

• I've updated the URLs of the APIs to match Kibana API naming conventions.

Please review all of the above and let me know if it makes sense to you.

[exekias]

This is looking good! A few notes:

• POST | /api/beats/agent/{beat UUID} | * Request should include Beat's public key, encrypted with shared secret <-- no need to encrypt the public key
• Why do you want to keep beat info after disenroll? not saying it's wrong, I'm just curious. We should think on the case of disenroll + enroll again

I noticed beats is back to the URL, is this because of the convention? I kinda liked to keep it out just in case we want other projects to use the enrollment part. It's ok if it must be like this

[ycombinator]

POST | /api/beats/agent/{beat UUID} | * Request should include Beat's public key, encrypted with shared secret <-- no need to encrypt the public key

So the reason I thought we should do this is to avoid this attack scenario:

• The Beat enrolls with Kibana. During this step the Beat sends its public key unencrypted to Kibana.
• A MITM attacker holds on to the public key and lets the enrollment request go through as-is to Kibana.
• At some point in the future an admin verifies the Beat in Kibana.
• In all future management API responses, the MITM attacker can now impersonate Kibana and change the payload + signature in responses from Kibana -> the Beat, since the attacker knows the Beat's public key. The Beat will happily verify the payload + signature using the private key, without realizing that those have been tampered with.

But it's possible I'm misunderstanding something in the flow so I'm happy to be corrected and simplify the design!

Why do you want to keep beat info after disenroll? not saying it's wrong, I'm just curious. We should think on the case of disenroll + enroll again

Not the beat's info like its IP or public key; those we can delete. Only keep its identity (UUID) + tag associations, effectively keeping around any configuration the user has made for the Beat. Otherwise a user will have to redo all the configuration for that Beat from scratch once the Beat is re-enrolled.

Of course, if the same Beat presents a different UUID when it re-enrolls, that's a different story. But here I'm thinking of the case where a Beat has to be disenrolled + reenrolled due to security reasons.

I noticed beats is back to the URL, is this because of the convention?

Yeah, its a convention in Kibana to separate all the different Kibana plugins' APIs from each other and avoid collisions in naming. The convention is to start the URL with /api/, then the id of the plugin (in our case that would be beats), then the rest of the API URL.

[exekias]

Thanks for the explanations.

I agree with the security concerns, problem is that we cannot consider the enrollment token safe, as it may be leaked. Let's give this a second thought, worst case scenario we need to rely on SSL security to trust the Kibana instance.

[ycombinator]

worst case scenario we need to rely on SSL security to trust the Kibana instance.

I looked at what APM does, and this is indeed what they recommend in their docs: https://www.elastic.co/guide/en/apm/server/6.2/security.html#secret-token

At this point, in order to move forward for an MVP, I think we should keep things simple. Concretely, I think we should go back to your original proposal of enrollment tokens and access tokens being issued (and later verified) by Kibana and recommend in our docs that the user must enable SSL/TLS on their Kibana server. WDYT?

[exekias]

Works for me 👍

[ph]

closing this issue, current CM implementation has diverged.