Define permissions user for fleet-server needs

[ruflin]

A username / password needs to be passed into the fleet-server to work. It needs to be defined what the permissions are that are required for this user.

For reference, the current permissions of the fleet-user can be found here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/setup.ts#L140

[ruflin]

@aleksmaus Would be great if you could take this on.

[ruflin]

@scunningham If you have some cycles, would be great if you could also help with this one to define the minimal permissions we need. We should also decide if we need a built in role in Elasticsearch for this or not.

[scunningham]

Specifying the required perms is the easy part.

• read/write on all of the associated fleet indices
• manage api key perms to create new api keys
• perms for application level api key to talk to fleet:

  fleet-server/cmd/fleet/schema.go

  Line 25 in bd7527d

  const kFleetAccessRolesJSON = `

• superset of perms to create a default api key:

  fleet-server/cmd/fleet/schema.go

  Line 38 in bd7527d

  const kFleetOutputRolesJSON = `

Note that the index permissions currently requested for the default api key are already too broad and should be narrowed to "create_doc", which limits the permission to read/only.

The hard part is how to accomplish this. Ideally, we would assign an apikey per installed fleet server. This is problematic for upgrade in that if we need to add more permissions for a newer version, we would have to generate a new api key for the targeted upgraded fleet server(s). There's no way to add permissions to an existing apikey as far as I can tell.

It would be easier to create a fleet-server-role which contains the superset of the permissions required for the current set of executing fleet servers. Unfortunately, the role would be bound to a corresponding user, and we would have to distribute username/password to each of the Fleet Servers. If any one of the Fleet Servers is popped, the password would need to be changed breaking all of the other servers in the field; not just the popped server.

Thoughts @ruflin @aleksmaus @blakerouse? User/PW is easier to implement, but not ideal from a security standpoint. We could, as an alternative, require a new api key to be generated for each fleet server as part of the upgrade process. We should discuss had that would work in practice. How are upgrades going to roll out; is there a corresponding Kibana upgrade that must happen first? It seems this would be an issue whether we use apikey's or user/pw as in somebody in the system would need to upgrade the role associated with the fleet-server-role.

[ruflin]

• Do we need write permissions on all fleet indices? What about the enrollment keys, does fleet server need write permissions? Also some .ds-* are in this list which I assume we don't need?
• For handling upgrading permissions, agree not possible today. @bytebilly are working on fixing this but not sure if it arrives in time. In general I like the idea that each fleet-server could have its own API Key.
• Could we go first with username / password + role and migrate later to the API Key part to unblock us? Question is how such a migration would look like later on? Ideally the permissions of the api key would be updated / refreshed and no new API key would be needed on upgrade.

[scunningham]

• We will have to use username/password in the short term. As has been pointed out, Fleet must create API keys during enrollment and currently an API key cannot be used to authenticate the creation of another API key with any useful privileges (see docs

• The fleet server user will need, in addition to ability to create api keys with the appropriate permission, access to the following indices:

Index	Perms
.fleet-actions	r
.fleet-actions-results	r/w
.fleet-agents	r/w
.fleet-enrollment-api-keys	r
.fleet-policies	r/w
.fleet-policies-leader	r/w
.fleet-servers	r/w

• The role descriptors for the default Fleet output API Key should look like this:

{
	"fleet-output": {
		"cluster": ["monitor"],
		"index": [{
			"names": [
				"logs-*",
				"metrics-*",
				"traces-*"
			],
			"privileges": [
				"create_doc",
				"indices:admin/auto_create"
			]
		}]
	}
}

However, we will need to test this. It is likely that the beats are expecting higher privileges. A fallback would have more privileges, but would be less secure:

{
	"fleet-output": {
		"cluster": ["monitor"],
		"index": [{
			"names": [
				"logs-*",
				"metrics-*",
				"traces-*"
			],
			"privileges": [
				"write",
				"create_index"
			]
		}]
	}
}

This is not ideal as it allows arbitrary write across any of the above outputs in the system. We should try to avoid this. "create_index" should not be necessary as an ILM controlled data stream should already be in place as part of the package install.

[nchaulet]

We probably need auto_configure too to create the datastream from the index template

[scunningham]

@nchaulet Agreed. Modified above.

[ph]

This is indeed good timing we were looking into limiting the permissions on the API key in elastic/kibana#89713

Do we really need auto_configure ? I though append was taking care of that? cc @skh for info.

I've thought we had another permission than create_doc that was not allowing us to update mapping but I can't find it in the current documentation.

[scunningham]

To follow, after conversation on #es-core-features, two items:

1. create_doc is effectively append_only when working with data streams:

<<
Dan Hermann 12:23 PM
ok, create_doc does not permit updating mappings on data streams. we made that change for precisely the use case you are describing.

2. We should revert to the same undocumented priv Kibana was using instead of "auto_configure":

<<
Sean Cunningham 12:23 PM
oh fantastic.
12:24
would we need the “auto_configure” priv as well or is that a by product of the “create_doc” priv?

Dan Hermann 12:26 PM
create_doc does not include auto_configure if that's what you're asking.

Sean Cunningham 12:27 PM
I guess I’m asking if an index request will fail if the endpoint does not have ‘auto_configure’ privs, and the target index for the data stream is not created yet.
12:28
Ie. is ‘create_doc’ the minimal privilege that we can safely grant to the endpoint/beat modules.

Dan Hermann 12:30 PM
yes, it will fail to create the index if it does not have auto_configure. I don't recall why we bundled auto-create and auto-put-mapping together in the auto_configure role. You could specify create_doc and auto-create without auto-put-mapping if that's what your particular use case needs.

Sean Cunningham 12:31 PM
is there a separate auto_create priv? I don’t see it in the docs. Or is that something we’d have to add.
New

Dan Hermann 12:35 PM
you can specify privileges directly without referencing roles (e.g.: indices:admin/auto_create for auto-create). I don't know that we suggest that often, but it works. if that's what you need for your use case, I would want to go back and remind myself why we tied auto-create and auto-put-mapping together in the auto_configure role

Sean Cunningham 12:36 PM
Perfect. Thank you. That should do the trick.

I'll update the mappings above appropirately.

[scunningham]

Also, the "cluster:monitor" privilege is too broad. We should consider removing it and see what it breaks. For one thing, the license api requires it. Do we really need it?

[ph]

I was discussing with @scunningham over the "monitor" cluster privileges and the beats, IIRC it is used for one or two things:

• Retrieving the license information from the remote cluster.
• Ping to the remote cluster.

The monitor permissions give a lot more information:

All cluster read-only operations, like cluster health and state, hot threads, node info, node and cluster stats, and pending cluster tasks.

As far as I know, there is no way to limit to only retrieve the remote license.

@urso @andresrc Would we consider removing the license check?

[danhermann]

Elasticsearch issue to track the addition of an append-only role: elastic/elasticsearch#68414

[ruflin]

I'm also wondering if we could skip the license check as fleet-server will be only for future releases?

[ph]

@scunningham is "events" a typo in #60 (comment) I believe this should be traces?

[scunningham]

@ph The permissions above were snapshotted out of kibana when I wrote original fleet off of the 7.9 branch.

The latest perms in Kibana are here

Looks like events have been replaced with traces.

const key = await createAPIKey(soClient, name, {
    'fleet-output': {
      cluster: ['monitor'],
      index: [
        {
          names: [
            'logs-*',
            'metrics-*',
            'traces-*',
            '.ds-logs-*',
            '.ds-metrics-*',
            '.ds-traces-*',
            '.logs-endpoint.diagnostic.collection-*',
            '.ds-.logs-endpoint.diagnostic.collection-*',
          ],
          privileges: ['write', 'create_index', 'indices:admin/auto_create'],
        },
      ],
    },
  });

I will update above.

There's also a bunch of suspect indices up there. Can anybody chime in as to why we have ".ds-" and ".logs-"? @nchaulet

[ruflin]

.ds-* is historically as data streams initially also required permissions for the indices itself. This is not needed anymore and should be removed from the Kibana code (an issue for it is somewhere). For the .logs this comes from endpoint telemetry. @kevinlog knows more here.

[ph]

@scunningham The .ds- will be ne removed, they were required because of an issue with data stream and the permission on the backing indices.

[kevinlog]

I spoke with @scunningham

Currently, when the Endpoint package is installed, it creates the index template, but the data stream isn't created until the Endpoint actually sends the first document, so this permission was required.

If we can create the data stream before the Endpoint sends the first document (i.e. the Endpoint no longer needs to be able to created the data stream, just send docs to it), then we should be OK to remove.

[ruflin]

Closing for Phase 2. Further follow up discussions needed.