Metadata for API keys

[bytebilly]

Description

We are considering to extend Elasticsearch API keys in order to support internal requirements by other teams, and avoid building new features that would increase complexity and maintenance costs.

The current implementation is a good starting point, but hits its limits in complex scenarios where those keys are used in a more sophisticated authentication flow.
At the moment, the solution is to build an external logic around keys, or reimplement an independent similar feature.

Proposal

One thing that could be very useful is to allow custom arbitrary metadata to be attached to an API key during the creation action. This is totally transparent to Elasticsearch, that has no knowledge of the meaning (and format) of the metadata, and doesn't perform any action on it.

An example of metadata could be the scope of the key, a description, or a cryptographic signature.

This implies that metadata can be passed to the Create API key call, stored as an attribute of the newly generated key, and then returned by the Get API key.

Each user can define the metadata, and build some logic around it.

Related Kibana issue: elastic/kibana#93820

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[bytebilly]

@arisonl @roncohen @tvernum as we discussed in previous conversations, this could be a very turning point to simplify the authentication and authorization flow for Ingest, where we actually use a separate service.

I'd like to keep it very generic, so that Elasticsearch has no knowledge of the metadata content, and it can be used in very different scenarios.

This is probably the only requirement on the Elasticsearch side to make API keys suitable for Ingest needs, but I'll continue to investigate about that.

[sorenlouv]

allow custom arbitrary metadata to be attached to an API key during the creation action

+1. This is needed for elastic/kibana#77966 (currently blocked until we can do something like that).

Use case
For APM we want to make it easier for users to create API keys that are used between an APM agent (eg the java agent) and the APM Server.
Currently we can create API keys, but have no way of retrieving them again since we cannot annotate API keys with apm or similar. We might also want to annotate the API key with additional metadata like a description, the environment etc.

[tvernum]

@sqren

I would have expected that all of those API keys would be owned by the same user (apm_system ?) and you would be able to retrieve them that way. Is that not the case?

[graphaelli]

Jumping in as @sqren is on vacation right now. Can you clarify whether the question is; are all API keys owned by a single user? Following on, if they were, would that user only have API keys used for agent/server auth?

Currently, API keys used for this purpose are not owned by a single user, they're not even created via kibana at the moment (see elastic/kibana#77966 for details).

[tvernum]

Shouldn't they be owned by a single user? Do you intentionally want APM API Keys to be owned and managed by individuals?

[simitt]

Right now there is no UI for creating API keys, non-cloud users can use a CLI tool built into the APM Server, cloud users need to follow the docs for creating the API key manually.
The CLI tool is beta and creates the API Keys via the provided user. The name of the apm_system user is a bit misleading as it has been created and used for writing monitoring data, aligned with the beats_system user. I don't think that user would necessarily be the right one to use here, at least we would need to give it additional privileges and update the docs accordingly.
We honestly also didn't consider creating/using a built in APM user for this. Which advantages would you see for creating this by a system user rather than an individual user?

[sorenlouv]

Did we reach a decision here?

I would have expected that all of those API keys would be owned by the same user (apm_system ?)

As mentioned above this is not the case.

Do you intentionally want APM API Keys to be owned and managed by individuals?

As I understand, when creating an API key via Kibana the request can either be made as the end user (in which case the API key will be owned by this user) or by the internal user (the shared kibana user).
I'm not sure the internal user is authorized to create API keys and we'd still have the problem of retrieving only APM specific keys.

I'm fine with letting individuals own/manage api keys unless there is a better approach.

[tvernum]

Did we reach a decision here?

Not yet. I don't think generic metadata is what you need in order to solve the attribution problem you have. I think we need something more specific around the owning "application" and potentially a reference id.

Which isn't to say that we won't add generic metadata, I just don't think it's helpful to solve common problems like this in ad-hoc ways via metadata.

I'm fine with letting individuals own/manage api keys unless there is a better approach.

The current design for API keys treats them as secondary, restricted, credentials for the owning user. For example, it an action is performed by an API key, the audit logs will record it as having been taken by the user that created the API key. Likewise, the _authenticate API will return the owning user's username. There are a lot of assumptions built into Elasticsearch that API keys operate on behalf of the user that created them.

I don't think that's what you want.

We don't currently have a mechanism for users creating API keys for use in/by other services. Our expectation was that in situations like this the API keys would be owned by whatever user APM server uses to write to Elasticsearch - that APM itself would own whatever API keys it needed.
I appreciate why that's tricky given the way APM works, and how APM server runs, but it's currently a mismatch between what these API keys were designed for, and the problem you're currently trying to solve.

[sorenlouv]

if an action is performed by an API key, the audit logs will record it as having been taken by the user that created the API key [...] I don't think that's what you want.

Correct, this doesn't sound great.

We don't currently have a mechanism for users creating API keys for use in/by other services

Okay, that makes sense. I'll sync with the rest of the team and we'll figure out what we want to do in the near future.

[graphaelli]

Which isn't to say that we won't add generic metadata, I just don't think it's helpful to solve common problems like this in ad-hoc ways via metadata.

I agree. I've opened #67390 as an alternative proposal.

[ywangd]

We have done some scoping and design on this feature and I'd like to share the current plan:

NOTE: Metadata on API keys are Not secret. Users who have access to view API keys (e.g. manage_api_key) will be able to view the metadata for any API keys. There will also be no way to reserve any metadata key or value for certain applications. Any user who can create API keys can set any metadata at will. Hence the metadata information cannot be relied upon by itself, e.g. it is wrong to assume a key belongs to some user purely based on its metadata value. The API key identity and ownership still must be checked for this purpose.

1. The CreateApiKey and GrantApiKey APIs wil be updated to accept an new metadata field in the payload. This field takes map of abitrary key (string) /value pairs (in java terms it is Map<String, Object>)
2. The new metadata field is of the flattened field type
3. The metadata will be returned in the response to the GetApiKey request.
4. The metadata will be available as part of the authentication object (via ThreadContext). This makes it possible to be accessed in ingest pipelines, in the future, specifically the SetSecurityUser processor. We still need to decide on what the configuration would look like for this new information.
5. The API key metadata will not be returned in the response to the authenticate API.
6. Invalidated and expired API keys are permanently deleted every so often (retention is up to 7 days). When the keys are deleted, the metadata associated with them are gone as well. There will be no way to retrieve any information about a key once it is deleted. Attempting to authenticate with a deleted key results in a 401 error.
7. The metadata will not be searchable initially. We do plan to support it. But it is considered as a separate piece of work and will be tracked with a different issue.
8. The metadata will not be updatable. Mutable API keys is of its own topic and updatable metadata should be tackled as part of it elsehere.

Here are some examples of what the APIs would look like after the proposed changes:

• CreateApiKey with metadata

PUT _security/api_key
{
 "name": "key-1",
 "role_descriptors": {
   "agent": {
     "indices": [
       {
         "names": [
           "index"
         ],
         "privileges": [
           "create_doc"
         ]
       }
     ]
   }
 },
 "metadata": {
   "application": "fleet-agent",
   "host-tag": "foo",
   "codes": [
     3,
     5,
     7,
     9
   ],
   "environment": {
     "os": "Linux",
     "level": "untrusted",
     "rating": 3
   }
 }
}

The response will be the same as what we currently have:

{
 "id": "Nm_y63cBLN0wxfKLwdEc",
 "name": "key-1",
 "api_key": "Mw7-sm8mTLWzk-XvKNiBWw"
}

• GetApiKey will return the metadata or {} if the key has no metadata (due to either no metadata was provided at creation time or old keys created before the metadata feature)

{
 "api_keys": [
   {
     "id": "MG_c63cBLN0wxfKLA9Ht",
     "name": "key-1",
     "creation": 1614569800692,
     "invalidated": false,
     "username": "elastic-admin",
     "realm": "file1",
     "metadata": {
       "host-tag": "foo",
       "codes": [
         3,
         5,
         7,
         9
       ],
       "environment": {
         "os": "Linux",
         "level": "untrusted",
         "rating": 3
       },
       "application": "fleet-agent"
     }
   },
   {
     "id": "Mm_g63cBLN0wxfKLMNGv",
     "name": "key-3",
     "creation": 1614570074288,
     "invalidated": false,
     "username": "elastic-admin",
     "realm": "file1",
     "metadata": {}
   }
 ]
}

[graphaelli]

Not stated explicitly, but there will be no method for updating metadata as part of this plan, correct?

Does the comment in #48182 (comment):

Which isn't to say that we won't add generic metadata, I just don't think it's helpful to solve common problems like this in ad-hoc ways via metadata.

still stand in regards to finding api keys created for an application. Put another way, can we close #67390 now and expect to use the future search capabilities on metadata for this purpose?

[ywangd]

Not stated explicitly, but there will be no method for updating metadata as part of this plan, correct?

Good catch! No it will not be updatable. I will add this to the plan to make it explicit. Generally we feel mutable API key is a broader topic that we'd like to tackle in a whole instead for just metadata.

Does the comment in #48182 (comment):

Which isn't to say that we won't add generic metadata, I just don't think it's helpful to solve common problems like this in ad-hoc ways via metadata.

still stand in regards to finding api keys created for an application. Put another way, can we close #67390 now and expect to use the future search capabilities on metadata for this purpose?

I think it could still be an issue for APM's particular use case. Metadata key/value cannot be reserved for any clients, i.e. anyone can write any metadata as will as long as they can create an API key. This means, the metadata {"application": "apm"} may not be unique to APM related API keys because I can also create such metadata accidentally or intentionally (even maliciously). Therefore relying only metadata to find all API keys belong to APM is not going to be robust. (I will add a section about this to the plan as well. Thanks)

[tvernum]

expect to use the future search capabilities on metadata for this purpose

It's not mentioned in @ywangd's original summary above, but we've discussed this elsewhere - one of the design choices we're proposing is that there will not be any security controls around setting metadata. That is, if you can create an API key then you can set any metadata you want on it.

Consequently, searching across all API keys in a cluster with metadata.app_name = "apm" will tell you which API keys were tagged by the creator as being "APM", but you can only trust that as representing some truth if you also trust that creator not to lie.

So, performing that search on API keys that were created by a user that you control would be reasonable. Performing that search in a cluster that is locked down to only be accessible by trusted users (e.g. the security cluster in ECE) is also reasonable. But searching across all creators on an arbitrary cluster where any number of users have the manage_own_api_key privilege is at risk of giving you API keys that aren't actually used for APM.

Whether that risk is acceptable depends on what you want to do with that list. If it's just shown in a UI then maybe it's OK, but it definitely not ideal and it's not a good option in general.

[bytebilly]

NOTE: Metadata on API keys are secret. Users who have access to view API keys (e.g. manage_api_key) will be able to view the metadata for any API keys.

@ywangd I'm not sure to get what you mean with secret. Seems like the opposite to me, as anybody that can manage API keys can also see them.

[ywangd]

NOTE: Metadata on API keys are secret. Users who have access to view API keys (e.g. manage_api_key) will be able to view the metadata for any API keys.

@ywangd I'm not sure to get what you mean with secret. Seems like the opposite to me, as anybody that can manage API keys can also see them.

🤦 🤦 I missed the critical word here. It should have been "not secret". I updated the original comments. Thanks for catching it.

[graphaelli]

Now that the initial support is in where should we track the additional work for this effort? We're primarily interested in searching metadata for elastic/kibana#77966.

[ywangd]

@graphaelli I created following two more issues to track feature work on API key metadata

• Search and Pagination for API keys #71023 Searching on API key metadata
• Expose API key metadata for ingest processors #71024 Expose API key metadata to ingest processors

Also note that the backport (#70956) for API key metadata is currently on hold and waiting for an agreed change on the system indices (@jaymode). We are still targeting for 7.13. But you may need test it against the master branch for the time being.