Custom privilege definition on roles

[elasticmachine]

Original comment by @jaymode:

In order to support security for other components of the stack, we will provide a primitive on the roles that exist in elasticsearch. This primitive is a new entry or entries alongside the current indices and cluster entries. The entries would contain an array of arbitrary string values.

If other applications in the stack need the ability to get a view of the current user's combined privileges, we would need an API to render this combined view. When merging roles we should simply combine all of the lists.

[elasticmachine]

Original comment by @skearns64:

If other applications in the stack need the ability to get a view of the current user's combined privileges, we would need an API to render this combined view

I'm +1 to an "effective privileges" API (LINK REDACTED), but would it be possible to also integrate these custom privileges with the has_privileges API?

[elasticmachine]

Original comment by @joshbressers:

After a chat with the Kibana folks they're going to want this feature to enable Spaces properly (they can live without OLS for now). We will probably want to bump the priority of this feature.

[elasticmachine]

Original comment by @jaymode:

@epixa do you have an idea of what you would need from us in terms of API/capabilities?

[elasticmachine]

Original comment by @epixa:

@jaymode At this point, I think we just need the ability to specify new privileges as strings and then have those privileges assigned to roles and returned via the has_privileges api.

[elasticmachine]

Original comment by @tvernum:

@epixa (and @jordansissel / @andrewvc if this is going to be useful to Logstash)

I've started implementing this, and there are 2 potential features that have a big impact on the implementation complexity. They're both achievable if they're truly useful (I have a prototype already), but I don't want to do them unless we actually need them.

1. pre-defined privileges
2. privileges on resources

pre-defined privileges

We have a choice, either privileges are just a string, or they are pre-defined objects.

I'll do my best to explain as succintly as possible.

If we have "just a string", then a role can have:

"custom": [ "kibana:read", "kibana:write" ]

And that's easy. But it means that "kibana:write" doesn't imply "kibana:create" because they're just strings, and they don't mean anything inside ES.

So

GET /_xpack/security/user/_has_privileges
{
   "custom": [ "kibana:write", "kibana:create" ]
}

would return

"kibana:write": true,
"kibana:create" false

And, likewise, kibana:all doesn't mean anything special to ES.

So, if Kibana (or Logstash, etc) wants to have a rich set of privileges like all implies read+write and write implies create+update+delete, then there would be 3 choices for implementing that on top of the "just a string" model, none of which are great.

1. expand them at role creation time, so if a role is supposed to have "kibana:all", you actually write it with

   "custom" : [ "kibana:all", "kibana:read", "kibana:write", "kibana:create", "kibana:update", "kibana:delete" ]
   

   and then when Kibana 7.0 adds a new "kibana:admin", you find all the roles with kibana:all and add kibana:admin into the list.
2. implement the implies logic at read time, so if you want to check "delete" access, you need to ask:

   GET /_xpack/security/user/_has_privileges
   {
       "custom": [ "kibana:all", "kibana:write", "kibana:delete" ]
   }
   

   and then the user has "delete" if it any of those privileges are true.
3. try and solve it with patterns instead of names. Instead of storing "kibana:all", you store "kibana:*" and "write" becomes "kibana:write/* and delete is "kibana:write/delete". Then ES can handle the all implies write implies delete logic by evaluating Automata, which we already support. But it means roles have seemingly artibrary magic patterns in them, and every kibana privilege needs to be planned out to fit into a perfect tree.

All of those end up trying to replicate functionality that ES already has for cluster and index privileges, because we treat those privileges as objects rather than strings.

We can do that for custom privileges, but it requires creating a full API to CRUD a custom privilege, and it means that resolving a user's custom privileges depends on reading them in from security index, so it's not just implementation effort, it's also runtime complexity.
But it would mean you could do something like

PUT /_xpack/security/privilege/kibana:all
{   "actions": [ "kibana:*" ] }

PUT /_xpack/security/privilege/kibana:read
{   "actions": [ "kibana:read/*" ] }

PUT /_xpack/security/privilege/kibana:write
{   "actions": [ "kibana:write/*" ] }

PUT /_xpack/security/privilege/kibana:delete
{   "actions": [ "kibana:write/delete/*" ] }

And then

GET /_xpack/security/user/_has_privileges
{
    "custom": [ "kibana:delete" ]
}

would work correctly for users with "all", "write" or "delete"
as would

GET /_xpack/security/user/_has_privileges
{
    "custom": [ "kibana:delete/dashboard" ]
}

I've elected to use a tree based model for custom actions there because it's worked well for ES, and it seems logical to do the same thing for custom privileges if we go down the pre-defined objects path, but it's certainly open for discussion.

privileges on resources

Cluster privileges are all or nothing - you have "monitor" for the whole cluster, or none of it.
Index privileges are tied to a resource - you have "read" for index-a, and "write" for index-b, but nothing on "index-c".

I can make custom privileges work like cluster privileges, or like index privileges (with the latter being added complexity).
Modeling a whole-application permission with a resource-bound privilege is easy, you can just assign the * resource and it will behave correctly (although it's could lead to confusion with customers).
It's very hard to get resource-level permissions using non-resourced privileges, so if you need them, ES should support them properly, but we get a simpler model if we can avoid it.

Side Note: Naming

During my prototype, I found myself getting the words "cluster" and "custom" confused a lot in my typing. I'd like to name "custom" privileges something else. My suggestions would be (in order of my preference)

• application
• external
• bespoke

[elasticmachine]

Original comment by @epixa:

cc @kobelb as he'll be doing or coordinating most of the work around this stuff on Kibana's end.

Implementation

Kibana will need the ability to manage a role hierarchy, and I think from an end-user perspective it should behave similarly to ES.

it's also runtime complexity.

Other than the existence of additional REST endpoints, how does the complexity of this implementation differ from the way ES privileges are handled today? It seems like we need to implement the complexity somewhere one way or another, so to me it's a question of whether we do it in ES or in Kibana, so understanding exactly what would be new complexity to ES is important here.

privileges on resources

This is a little tricky for Kibana. On the surface, we want kibana-wide privileges which are sort of like cluster level privileges. There wouldn't be any concept of "write" for .kibana but only "read" for .reporting, for example. You'd simply have the "write" privilege for all of kibana.

The challenge here is that some people run multiple different kibana apps on a single cluster. I can only think of one way to support this: we do index-level permissions and then hang all of the permissions for a single kibana install off of its corresponding .kibana index. Kibana then always check privileges on its index regardless of whether it ultimately accesses somewhere else like .reporting.

I'm open to ideas here, though!

Naming

I don't have strong opinions about this, but if we do cluster-level privileges, calling them "application" privileges doesn't seem accurate. If we did index level privileges or if ES somehow tracked the notion of external applications, then application privileges would make a lot of sense to me. External privileges seems fine either way.

[elasticmachine]

Original comment by @kobelb:

Court's done a good job, like usual, summarizing Kibana's usage/needs.

The challenge here is that some people run multiple different kibana apps on a single cluster. I can only think of one way to support this: we do index-level permissions and then hang all of the permissions for a single kibana install off of its corresponding .kibana index. Kibana then always check privileges on its index regardless of whether it ultimately accesses somewhere else like .reporting.

I think this makes sense, and I can't think of a better alternative. The only other option I can think of would be to keep the reporting custom privileges/roles in the .reporting index, but then we'd have to enumerate all of the application indices when allowing users to choose custom roles for users, making this process more complicated. I don't see any downsides to putting all of these roles/privileges in the .kibana index that corresponds to the .reporting index.

[elasticmachine]

Original comment by @tvernum:

Other than the existence of additional REST endpoints, how does the complexity of this implementation differ from the way ES privileges are handled today?

The main difference is that the existing privileges are all fixed in code, and available with zero-lookup.

API-defined privileges need to be stored in an index (technically there's other places, but it would be an index) which means we need to do I/O and async lookups and deal with unavailable shards, etc.

All of which is totally fine - we do it for users, roles, role mappings, etc etc. But it's overhead that we don't want to pay for if no one needs it.

[elasticmachine]

Original comment by @kobelb:

The main difference is that the existing privileges are all fixed in code, and available with zero-lookup.

API-defined privileges need to be stored in an index (technically there's other places, but it would be an index) which means we need to do I/O and async lookups and deal with unavailable shards, etc.

All of which is totally fine - we do it for users, roles, role mappings, etc etc. But it's overhead that we don't want to pay for if no one needs it.

It'll definitely be used by Kibana, and would provide a more consistent approach, so I'd also prefer it was added on the Elasticsearch side of things.

[elasticmachine]

Original comment by @tvernum:

The challenge here is that some people run multiple different kibana apps on a single cluster. I can only think of one way to support this: we do index-level permissions and then hang all of the permissions for a single kibana install off of its corresponding .kibana index. Kibana then always check privileges on its index regardless of whether it ultimately accesses somewhere else like .reporting

In my examples above, I name spaced all the Kibana privileges, e.g. kibana:read (or maybe kibana:reporting is more representative). I plan on enforcing that external-bespoke-application privileges have an application prefix.
If you had some system for enforcing unique "kibana-name" across the kibana instances, the you could put the privileges into separate namespaces - kibana-sales:reporting, kibana-soc:reporting etc.
I suspect that would get messy with renaming, and uniqueness. But it would make version upgrades easier (upgrading 1 kibana wouldn't change the privilege set for another kibana).

[elasticmachine]

Original comment by @clintongormley:

I think before we can answer a question about how to structure these privileges, we should make a big list of all the privileges that we need. The answer should fall out of that

[elasticmachine]

Original comment by @epixa:

We don't know all of the privileges we'll need, but I can give an example that includes some that we do know. For the sake of not conflating our needs with any specific implementation, I'm just going to describe intentions here rather than actually giving each privilege a specific name. Nesting is represented with indents, so to give access to all children, you'd only need to give access to the parent.

can access kibana (can they login and have a user profile?)
developer access (can they use dev tools?)
monitoring access (can they access monitoring?)
reporting access (own reports only, may not actually be necessary if we apply OLS to reporting)
management access (can they access the management app?)
    manage active user sessions
admin access
    read global kibana data (e.g. user profiles, system health, installed plugins)
    execute global kibana actions (e.g. changing settings)
    reporting admin
        can see all reports
        can delete any report

This is just a straw man. I can envision a totally different approach to privilege hierarchy that breaks down privileges by the [core_]plugins that they are relevant to alongside a single set of "global" admin level privileges. From an ES perspective, the two approaches are identical.

[elasticmachine]

Original comment by @kobelb:

If you had some system for enforcing unique "kibana-name" across the kibana instances, the you could put the privileges into separate namespaces - kibana-sales:reporting, kibana-soc:reporting etc.
I suspect that would get messy with renaming, and uniqueness. But it would make version upgrades easier (upgrading 1 kibana wouldn't change the privilege set for another kibana).

Just to clarify how this will likely work, when we create a specific role that has the custom Kibana privileges it will be scoped to specific indices, as this is the way that Elasticsearch roles work currently. By default we'd created the role that grants privileges to the default .kibana index. If the user wished to create a separate instance of Kibana that used a different index, they'd have to duplicate this role with the custom privileges and assign it to the different index.

Within Kibana's server, we'd be verifying that the user has a role with specific custom privileges on the index. By convention, we'd be requiring users to create these namespaces themselves.

[elasticmachine]

Original comment by @kobelb:

After the discussion at engineering all hands, @legrego and I discussed how custom privileges on roles could be used by RBAC, and it's something we'd like to use. Additionally, the ability to specify the privilege tree would be greatly beneficial.

[elasticmachine]

Original comment by @tvernum:

@kobelb I'll draft something about how I think you could model the privileges described in epixa's LINK REDACTED, and then we can flesh out the details.

[elasticmachine]

Original comment by @kobelb:

Fantastic, thanks @tvernum.

[elasticmachine]

Original comment by @jaymode:

I had a chat today with @AlexP-Elastic @swallez @zanbel about the future use of x-pack security in ECE. One thing that they are looking at is using x-pack roles as the basis for authorization; currently they are only doing it at the role name if I understood correctly. I asked them to take a look at this issue and provide feedback if this is something that they would envision using.

[elasticmachine]

Original comment by @jordansissel:

/cc @andrewvc @tsg @monicasarbu -- I don't think anyone from Logstash has chimed in yet about custom privilege needs. Background for Logstash: Having custom privileges may enable more powerful API in Logstash by allowing API calls to be authenticated by x-pack security in Elasticsearch. In the past, I have rejected requests for things like a step debugger, simulate api, etc, for lack of any security in the Logstash API.

[elasticmachine]

Original comment by @tvernum:

@kobelb Here's the draft proposal I promised

---

Background

Elasticsearch implements Privileges as a management layer on top of Actions. My proposal continues that model to custom privileges for 3 reasons:

1. That's the code we have, and part of the reasoning for building this inside ES is to take advantage of what we already have (working, tested, maintained).
2. It works quite well for ES
3. It mimics some of the systems that I've had the best experiences with in previous roles. One of the common mistakes that happen in systems that try to model & manage security rights is that there is tight coupling between the concepts that are used to on the management side (features that exist for customers to work with) and the concepts that are used to verify a user's access at runtime.

An example of where that goes wrong is with a naive implementation of roles, whereby

• Users have roles
• The security layer checks for the existence of a role before granting access to a function.

But if the role is security_manager, then the same runtime checks will exist in reset user's password , create user, and grant role to user.
But now if you want to implement an organisational separation where the Helpdesk can reset passwords, but only Security Operations can create users, you need break that apart and change the runtime checks to test for a new reset_password role that is separate to setup_users.

The problem is that the runtime security checks really ought to be testing for access to "functions" (aka "features", "capabilities", or "actions") rather than checking for specific roles.

In the ES security model, "Privileges" are on the user-management side, and "Actions" are on the runtime-checking side, and the linkage is that a Privilege defines all the Actions that it grants (with support for wildcards and regexp).

Actions

So, step 1 is to define the Actions that represented the Kibana features that need security checks.
As a constraint, I'm proposing that

1. All actions needs a namespace prefix that represents the app they belong to: e.g. kibana:
2. All actions need to have a / in them (to distinguish between action names and privilege names).

As a recommendation, actions should exist in a logical tree, separated by / so that it is simple to use wildcards to imply a set of related actions.

So, I took a stab at some proposed actions from what @epixa described, and what I know of Kibana:

kibana:/login
kibana:/tools/developer
kibana:/tools/monitoring
kibana:/tools/reporting
kibana:/tools/management
kibana:/manage/security/sessions
kibana:/manage/foo/bar
kibana:/admin/read/users/profile
kibana:/admin/read/system/health
kibana:/admin/read/system/plugins
kibana:/admin/write/settings
kibana:/admin/reporting/view
kibana:/admin/reporting/delete

I don't really know what the difference between manage and admin is, but it seems to fall out of the proposed privileges, so I just ran with it.
Perhaps /manage/security/session needs to move to be /admin/security/sessions.

This means that:

• someone with kibana:/admin/* has access to view user profiles, view system health, update settings, delete reports, etc.
• someone with kibana:/admin/read/* has access to view user profiles & view system health, but cannot update settings or access reports.
• someone with kibana:/admin/read/system/* has access to view system health & plugins, but cannot view user profiles, update settings or access reports.
• etc.

Privileges

Privileges are how we can group multiple action together into something that makes sense for a customer to work with.

The constraints I'm proposing for privileges names are:

1. They needs a namespace prefix that matches the actions they imply (so Kibana privileges can only grant Kibana actions)
2. Other than the prefix separator (:), they can only be composed of [a-zA-Z0-9_.-] (which distinguishes them from action names which must contains a /)

I don't really know what the right names/breakdown for these are, but I've taken a rough guess:

# A terrible name
kibana:basic = [ "kibana:/login" ] 
kibana:all = [ "kibana:/*" ]
kibana:developer = [ "kibana:/login" , "kibana:/tools/developer" ]
kibana:monitoring = [ "kibana:/login" , "kibana:/tools/monitoring" ]
kibana:reporting = [ "kibana:/login" , "kibana:/tools/reporting" ]
kibana:management = [ "kibana:/login" , "kibana:/tools/management" ]
kibana:security = [
  "kibana:/login" , "kibana:/tools/management", "kibana:/manage/security/*"
]
kibana:monitor = [ "kibana:/login" , "kibana:/admin/read/*" ]
kibana:configure = [
  "kibana:/login" , "kibana:/admin/read/*", "kibana:/admin/write/*"
]
kibana:reporting_admin = [ 
  "kibana:/login" , "kibana:/tools/reporting", "kibana:/admin/reporting/*"
]

Runtime checks

When you call the _has_privileges API, you'd typically pass in an action.
e.g.

GET /_xpack/security/user/_has_privileges
{
   "custom": [ "kibana:/admin/read/system/health" ]
}

If a user has kibana:all, kibana:monitor or kibana:configure then this will return true, because those privileges all imply the kibana:/admin/read/system/health action.

[elasticmachine]

Original comment by @kobelb:

When discussing this functionality previously, I was pushing for us to be able to define these custom privileges on specific indexes in Elasticsearch so we could scope the privileges to a specific instance of Kibana by specifying the appropriate .kibana index. After further reflection, this approach of custom privileges on indieces isn't something that we wish to do any longer and instead we'd like to utilize the approach that @tvernum outlined here:

If you had some system for enforcing unique "kibana-name" across the kibana instances, the you could put the privileges into separate namespaces - kibana-sales:reporting, kibana-soc:reporting etc.

From my understanding, using the namespacing approach would work nicely with your proposal. We'll have to figure out a way to get all of the custom privileges/actions into Elasticsearch, but that's a technical detail I'm sure we can figure out.

[elasticmachine]

Original comment by @kobelb:

I've been thinking through Kibana's usage of the custom privileges/actions, and how we'd initially create them and how upgrades would work. The way I'm envisioning it, the privileges themselves would be rather static and we'd only be adding privileges during a minor upgrade; however, I foresee us wanting to change the actions for the privileges themselves rather frequently as we add/remove/refactor functionality.

The following situation assumes that Kibana executes a series of PUT requests on startup to ensure the privileges and actions are persisted to Elasticsearch before we start up Kibana. The very first time that we deploy a version of Kibana that utilizes the custom privileges and actions, we can execute a PUT request for a custom privilege that is mapped to any number of actions. However, if we were to release a new version of Kibana that added to the list of actions, and one of the older instances of Kibana was to restart before we had a chance to upgrade all instances, it would overwrite the new list of actions.

The only thing that I've been able to come-up with to solve this situation is Elasticsearch's custom privileges/actions themselves implementing the concept of a "version" or the Kibana server code itself doing the privilege to action mappings (that way we can have multiple "versions" running concurrently).

Perhaps the other teams that are considering utilizing the custom privileges/actions have an alternate solution?

[elasticmachine]

Original comment by @clintongormley:

(that way we can have multiple "versions" running concurrently).

I think it would be a mistake to have two versions of kibana running against the same kibana index (and the same kibana privileges namespace). Not just from the privileges side, but from the kibana objects side too.

Instead, we should store a minimum version somewhere (perhaps kibana index? perhaps custom privileges namespace?), and earlier versions should not overwrite later versions.

[elasticmachine]

Original comment by @tvernum:

Elasticsearch's custom privileges/actions themselves implementing the concept of a "version

We can definitely support metadata on custom privileges - we have it for almost every other object type in security, so it's pretty easy to include here. But by itself that's not quite enough, you could still get a race condition if you always try and update-privileges-unless-they-are-newer-than-my-version.
I think you want to do what we do for the .security index and store the version and then update-privileges-if-older-than-the-kibana-version. Then the race condition should only come into play if a user starts up both 6.3.0 and 6.4.0 against an index that still had 6.2.0 privileges. But then they're asking for trouble.

[elasticmachine]

Original comment by @kobelb:

Instead, we should store a minimum version somewhere (perhaps kibana index? perhaps custom privileges namespace?), and earlier versions should not overwrite later versions.

I like Clint's suggestion here of storing the version of Kibana in the .kibana index and using it to verify whether we should perform the startup-up logic that inserts the custom privileges/actions. I could also see it being beneficial to prevent older versions of Kibana continuing to interact with the .kibana index after any other migrations have occurred that could result in various other interesting race conditions.

[elasticmachine]

Original comment by @kobelb:

After further thought, even if we were to check the version numbers against the .kibana index on startup, how would we ensure that multiple instances of Kibana with different versions wouldn't get past that check, both thinking they're the correct version, and then proceed to do the startup logic, leading to race conditions and inconsistent state?

[elasticmachine]

Original comment by @tvernum:

proceed to do the startup logic, leading to race conditions and inconsistent state

Make your startup logic idempotent?
I mean that as a serious answer. I don't imagine customers are running a dozen Kibana instances all pointing at the same .kibana index, so your best bet is to simply make it safe to run the startup logic twice. Then you get automatically retry-on-failure/timeout/etc.

[elasticmachine]

Original comment by @kobelb:

Make your startup logic idempotent?
I mean that as a serious answer. I don't imagine customers are running a dozen Kibana instances all pointing at the same .kibana index, so your best bet is to simply make it safe to run the startup logic twice. Then you get automatically retry-on-failure/timeout/etc.

That was the initial plan, until multiple versions came into play.

Assuming the following situation where we're currently running Kibana 6.2, and then there are two "new" instances just coming up 6.3 and 6.4. If both 6.3 and 6.4 were to get past the update/check version and then both execute their series of idempotent PUTs of the privileges and actions, we end up in an inconsistent state.

Even though this situation is rather unlikely, I'm not comfortable with it being possible if we're using the startup logic to essentially control who accesses what inside of Kibana.

[elasticmachine]

Original comment by @clintongormley:

The only way you're going to get around that is with locking, which then introduces the problem of refreshing and timing out the locks (combined with skewed clocks).

I think we can safely put this into the realm of PEBKAC and just tell people not to do it.

[elasticmachine]

Original comment by @kobelb:

If there isn't a good solution to synchronizing the privileges and actions into Elasticseach across versions, I'd rather keep the privileges in Elasticsearch but the actions that correspond to these privileges in Kibana source code.

The main advantage that we get from storing the actions and their mapping to privileges in Elasticsearch is the ability to call the "hasPrivilege" API in Elasticsearch and have ES handle the actions mapping to the actual granted privileges. However, if this comes at the expense of potentially allowing users to perform actions in Kibana that they aren't authorized to do so, it's really not worth it.

If we were talking about anything else besides security, accepting the PEBKAC would be just fine, but we're talking about authorization here.

[elasticmachine]

Original comment by @kobelb:

We might have a way to get around the mixed-versions problem... I believe there are restrictions in-place between which versions of Kibana can communicate with which versions of Elasticsearch, there's potential for us to utilize this functionality to prevent the scenarios that I've previously outlined. Let me track down whether this is a possible solution before anyone else wastes effort addressing this concern.

[elasticmachine]

Original comment by @kobelb:

The way that we currently implement the version restrictions on Kibana to ES communication won't help us in this situation, since we allow older versions of Kibana to communicate with newer versions of Elasticsearch. Additionally, this version check in Kibana is only done on startup and then periodically as part of our healthcheck, so there's a window of opportunity for any version of Kibana to communicate with any version of Elasticsearch.

My concerns have been previously expressed through a number of comments, so in an effort to communicate my concerns to generate potential alternate solutions, I'm going to attempt to summarize my concern below. Apologies to those of you who have been following along and have a grasp of the issue at hand, feel free to ignore the following.

In this proposal we will be storing all privileges and their mappings to actions in Elasticsearch itself. Borrowing from @tvernum's examples earlier, we could create the following privileges that are mapped to specific actions:

kibana:all = [ "kibana:/*" ]
kibana:management = [ "kibana:/login" , "kibana:/tools/management" ]
kibana:security = [
  "kibana:/login" , "kibana:/tools/management", "kibana:/manage/security/*"
]

These custom privileges and actions would be created within Elasticsearch through a series of PUT requests on startup, that themselves would be idempotent. Elasticsearch roles would then be created that are associated with these custom privileges, and users would be assigned to these roles.

Kibana would then determine whether a user was granted access to a specific action by executing a request similar to the following against Elasticsearch:

GET /_xpack/security/user/_has_privileges
{
   "custom": [ "kibana:/tools/management" ]
}

The crux of the problem comes down to us allowing different versions of Kibana to communicate with different versions of Elasticsearch, so when Kibana is upgraded that there is no downtime.

This creates a problem when synchronizing the privileges and actions to Elasticsearch on startup, and when different versions of Kibana are utilizing the _has_privileges API to determine whether a user is authorized to perform an action.

With regard to the problem of synchronizing the privileges and actions to Elasticsearch, if two instances of Kibana with different versions executed their startup logic at the same time we would have their series of PUT requests for the different privileges interleave, leading to an unpredictable state. @clintongormley suggested performing a check of the version of Kibana before executing the synchronization, so that we only executed the privilege and action PUTs when the version matches the version specified in the .kibana index, but this doesn’t work if two new versions of Kibana came on line at the same time and got past the version check and then began executing their series of PUTs. An alternate solution that doesn’t have this limitation is if we could execute a single PUT for all privileges and their actions, so the last one to execute the startup logic wins, but this brings us to the second issue.

To support upgrading Elasticsearch first and then upgrading Kibana, we currently allow older instances of Kibana to communicate with newer instances of Elasticsearch. This means that we’ll have multiple different versions of Kibana executing the “_has_privileges” check against the same instance of Elasticsearch. The actions that are tied to the different privileges will likely change for the different versions of Kibana, and as such each version requires this _has_privileges check to be run against it’s view of which actions are tied to which privileges.

If we want to store the actions to privileges mappings in Elasticsearch, I don’t see how we can get around having the privileges to actions mappings themselves be versioned. This could be done by adding a parameter when the privileges are inserted, specifying the version:

PUT /_xpack/security/privilege/kibana:all
{
  “version”: 1
  "actions": [ "kibana:*" ] 
}

And then when we check the actions, we would specify the version of the actions that we’re checking:

GET /_xpack/security/user/_has_privileges
{
  “version”: 1
   "custom": [ "kibana:/tools/management" ]
}

We’d want the roles to be associated with the “un-versioned” privilege, so that when we do upgrade the privileges to actions mappings, we wouldn’t be requiring the user to go through an mutate all of their existing roles.

If we only stored the privileges themselves in Elasticsearch, we wouldn’t have to do this versioning of the privileges to actions mapping, and instead we could do this in the Kibana server code and this would be versioned implicitly.

While I see there being benefit to us always performing authorization via the _has_privileges API in Elasticsearch across the stack, at this point I'd prefer to keep the privileges to actions mappings in Kibana and implement our own tree structure of privileges to actions. We're already going to have to map the privileges to actions in code in Kibana to synchronize them to Elasticsearch, so we might as well just consume these directly. There's obvious benefit in the privileges themselves being in Elasticsearch as they're tied to the roles themselves, but the actions that correspond to the privileges are a Kibana specific concern.

[elasticmachine]

Original comment by @clintongormley:

@kobelb We neither test nor support running multiple versions of kibana against the same index. It is expected that there will be some downtime when upgrading.

[elasticmachine]

Original comment by @kobelb:

@clintongormley we do for minors: https://github.com/elastic/kibana/blob/aa69744f67547484e3b80d122a6c83baa7ce4d80/src/core_plugins/elasticsearch/lib/__tests__/is_es_compatible_with_kibana.js#L27

This allows us to run Kibana 6.2.0 against ES 6.3.0

[elasticmachine]

Original comment by @clintongormley:

@kobelb that's different, that's about different versions of kibana and elasticsearch. The problem we started with was two different versions of kibana running against the same kibana index at the same time. That isn't supported. So if we store the kibana version in the index, we can check that version at startup time, problem solved.

[elasticmachine]

Original comment by @kobelb:

@kobelb that's different, that's about different versions of kibana and elasticsearch. The problem we started with was two different versions of kibana running against the same kibana index at the same time. That isn't supported. So if we store the kibana version in the index, we can check that version at startup time, problem solved

Just to be clear, we do support this now. I was just able to spin up an instance of Elasticsearch 6.2.0, Kibana 6.1.0 and Kibana 6.2.0 and they're all operating normally.

If we were to change this behavior, and store the version of Kibana in the .kibana index, we'd need some way to check this version in the .kibana index before executing the _has_privileges check against Elasticsearch and ensure the version doesn't change between when we check the version number and when we execute the _has_privileges check, otherwise we risk performing authorization against a set of incorrect actions. I'm unaware of how we'd do something like this without support for transactions across completely different indices and API calls.

If we were to use the Elasticsearch version itself to perform a similar function, we'd need some way to specify the expected version number in the _has_privileges check itself, and have Elasticsearch reject the request. However, this would negatively affect our ability to do rolling upgrades of Kibana because as soon as we upgraded Elasticsearch, all of the instances of Kibana would immediately become dead in the water.

[elasticmachine]

Original comment by @clintongormley:

Just to be clear, we do support this now. I was just able to spin up an instance of Elasticsearch 6.2.0, Kibana 6.1.0 and Kibana 6.2.0 and they're all operating normally.

The fact that it appears to work is not the same as supporting it. There are no tests, we have no forwards compatibility policy, and we document shutting down the old kibana before starting the new one: https://www.elastic.co/guide/en/kibana/current/upgrade-standard.html

we'd need some way to check this version in the .kibana index before executing the _has_privileges check against Elasticsearch and ensure the version doesn't change between when we check the version number and when we execute the _has_privileges check

If the user has the ability to install new versions of the kibana server (which includes providing the kibana user credentials), then what are we trying to defend against? Only user error. I think this is way out of scope. The product can only function within the stated rules: shut down the old server before starting the new one.

[elasticmachine]

Original comment by @kobelb:

The fact that it appears to work is not the same as supporting it. There are no tests, we have no forwards compatibility policy, and we document shutting down the old kibana before starting the new one: https://www.elastic.co/guide/en/kibana/current/upgrade-standard.html

Nowhere in those docs do we recommend shutting down all instances of Kibana across the entire Elasticsearch cluster before starting a single instance of the new Kibana version up. Those instructions appear to be largely targeted at upgrading a single instance of Kibana. The rolling upgrade of Kibana is something that I've been told that we support; however, I don't see it clearly documented anywhere. However, the way the code is written, it definitely appears to work that way. @epixa do you happen to know of any definitive documentation whether we support the rolling upgrade of Kibana or not?

If the user has the ability to install new versions of the kibana server (which includes providing the kibana user credentials), then what are we trying to defend against? Only user error. I think this is way out of scope. The product can only function within the stated rules: shut down the old server before starting the new one.

I'm trying to ensure that we're authorizing access to Kibana in a predictable and consistent manner. If starting up a single new instance of Kibana authorizes a user in an older version of Kibana to perform actions they previously were unable to perform, this seems like a serious security vulnerability.

[elasticmachine]

Original comment by @tvernum:

While this feature is a helpful example of the sorts of problems that can come up when trying to support (or prevent) mixed versions, trying to solve it here is not going to be effective.
This really needs to be a Kibana wide solution (for whatever you guys agree the requirements are). You won't have a lot of success if every engineer needs to think about a piecemeal solution for every separate feature.

If you support mixed versions, then it affects everything. Any feature could write a change to the Kibana index that is not backwards compatible with earlier Kibana versions. So, if there's a desire to officially support this, then it's a big task. Maintaining that sort of backwards compatibility in ES (to support rolling upgrades) takes a lot of time and effort. You don't want to embark on that journey without some serious consideration.

I assume that the team will conclude that it's not worthwhile to support mixed versions, in which case I have a solution for you.

We can definitely support a bulk privileges API to make this as atomic as possible (it won't actually be atomic, because ES doesn't work that way, but we'll minimise the window).

Then, just use a psuedo action to control version access. When you install the privileges, include

kibana:basic = [ "kibana:/login", "kibana:/version/6.4.*" ]

etc (but watch out for :all)

Then when you want to check privileges, include a check for kibana:/version/${this.version}
That will automatically lock out incompatible versions of Kibana.

[elasticmachine]

Original comment by @kobelb:

After talking with @epixa, we're alright with making it so that when a new instance of Kibana starts up it essentially makes all older instances running against the same cluster no longer authorize users, so the proposal that @tvernum just made solves that situation.

The fact that we're allowing this situation currently, with a Kibana 6.2 and 6.1 running at the same time against Elasticsearch 6.2 is indeed "not supported" as you were alluding to @clintongormley, it just happens to work somewhat, yet in unpredictable ways. Apologies for my misunderstanding earlier.

[elasticmachine]

Original comment by @legrego:

How do we want to assign custom permissions to roles? Will we modify the existing role API endpoints to include this, or will it be managed by a separate endpoint? I'm trying to stub out the ES implementation in Kibana, and I want to make sure I'm moving in the right direction

[elasticmachine]

Original comment by @tvernum:

There will be new APIs to create custom privileges, but we will use the existing roles API to associate those privilges with roles.

[elasticmachine]

Original comment by @kobelb:

@tvernum is it safe to assume that work is on-going on implementing the custom privileges/actions? I know that there has been a lot of "chatter" from the Kibana team in regard to our requirements, but it feels like this has stabilized and your initial proposal seems to satisfy our envisioned use-cases for RBAC.

[elasticmachine]

Original comment by @tvernum:

Yes, it's my plan to pick it up again with a target of 6.4.
I think we have enough idea of the required API to do that, but I don't want to merge it until we know that it's the right API for you.

It would be good to hear more from cloud EMAIL REDACTED / @swallez ) and ingest EMAIL REDACTED / @tsg) about whether this seems to suit their future needs, but I think we're implementing a full featured setup, so if it works for Kibana it probably works more generally.

[elasticmachine]

Original comment by @clintongormley:

@tvernum I know that @kobelb is wanting to use the custom privileges feature for work he's doing for the 6.4 release, so it would be good to come up with at least an API spec as early as possible so that they can work in parallel with you.

[elasticmachine]

Original comment by @tvernum:

@kobelb I've revived my prototype, and brought it mostly back into shape. I should have something you can play with later this week.

[elasticmachine]

Original comment by @kobelb:

@tvernum awesome, thanks so much!

[elasticmachine]

Original comment by @jaymode:

@tvernum would you mind adding a comment that describes the work you're doing? There is a lot of back and forth on this issue so it would be good to have a nice summary of the feature being implemented.

[elasticmachine]

Original comment by @tvernum:

Naming

I've called them "application privileges" because custom and cluster are too close and were bound to cause an implementation bug at some point where IDE auto-completion picked the wrong field somewhere.

Application privileges have an implicit explicit namespace.
The APIs that deal with application privileges expect an "application name".
Both the "name" and the "actions" (see below) need to start with the same application name, followed by a :, e.g. kibana:read and kibana:/admin/settings/delete

Application names must match the regex ^[a-z][A-Za-z0-9_-]{2,}$
Privilege names must match the regex ^[a-z][a-zA-Z0-9_.-]*$
Privilege names must have the prefix, followed by a sequence of [a-zA-Z0-9_.-]+

Privilege actions must contain a / , : or * to distinguish them from names.
Don't start an action with / because X-Pack security treats a leading / as indicating a (lucene) regex rather than a simple wildcard match.

Defining Privileges

Ther eare 3 new end points: GET, PUT, DELETE privilege

On my WIP they are:

• GET /_xpack/security/privilege to get all application privileges
• GET /_xpack/security/privilege/{application} to get all application privileges for a specific application
• GET /_xpack/security/privilege/{application/{privilege} to get a subset of privileges by application+name. This supports wildcards and a comma-separated list of privilege names.
• PUT /_xpack/security/privilege/{application}/{privilege} to put a single privilege by name.
• POST /_xpack/security/privilege/{application}/{privilege}, works as per PUT
• POST /_xpack/security/privilege/ to store multiple privileges in a single API call.
• DELETE /_xpack/security/privilege/{application}/{privilege} to delete one or more privileges. This supports a comma-separated list for the privilege name (but not wildcards). If you want to delete everything under kibana/ then you need to do a GET then DELETE. Wildcards in DELETE API is dangerous and I didn't think it was worth trying to support it.

For the POST of multiple privileges the JSON looks like:

{
  "kibana": {
    "login": {
      "application": "kibana",
      "name": "login",
      "actions": [ "action:login", "version:6.4.*" ],
      "metadata": {
        "kibana-version": "6.4.0"
      }
    },
    "read": {
      "application": "kibana",
      "name": "read",
      "actions": [ "action:login", "version:6.4.*", "data:read/*" ],
      "metadata": {
        "kibana-version": "6.4.0"
      }
    }
  }
}

Assigning Privileges

Roles gain a new "applications" field.

PUT /_xpack/security/role/kibana_read
{
  "cluster": [ ],
  "index": [ ],
  "applications": [ 
    { "application" : "kibana", "privileges": [ "read" ] , "resources": [ "*" ] } 
  ]
}

"resources" follow the same matching rules as index names, but their meaning is entirely up to the consumer. They could be used for Kibana objects if that's helpful, or you can just assign the * resource if you don't need the privilege to refer to a specific object/resource.

Checking privileges

The _has_privileges API gains an "application" field:

GET /_xpack/security/user/_has_privileges
{
    "applications": [
        { 
            "application":"kibana",
             "resources":["*"], 
             "privileges":[ "version:6.4.0", "read" ]
        } 
    ]
}

The "privileges" field can contain a mix of privilege-names (likeread) or action names (like version:6.4.0)
The implementation is the same as per the existing index behaviour in _has_privileges.
Resources and Privileges are both treated as wildcards (via lucene automata) and assigned privileges must full overlap the privileges to be tested. Application names must be fixed strings.
So:

• with privilege: { "application": "foo", "name": "read", "actions": [ "data:read/*" ] }
• and role: { "application": "foo", "privileges": [ "read" ] , "resources": [ "abc*" ] }

The results of _has_privileges are:

• { "application":"foo", "privileges":[ "read" ], "resources":[ "abc*" ]} - true
• { "application":"foo", "privileges":[ "read" ], "resources":[ "abcde*" ]} - true
• { "application":"foo", "privileges":[ "read" ], "resources":[ "abcdef" ]} - true
• { "application":"foo", "privileges":[ "data:read/name" ], "resources":[ "abc*" ]} - true
• { "application":"foo", "privileges":[ "data:read/name" ], "resources":[ "abcdef" ]} - true
• { "application":"foo", "privileges":[ "read" ], "resources":[ "aaa" ]} - false
• { "application":"foo", "privileges":[ "read" ], "resources":[ "a*cdef" ]} - false
• { "application":"foo", "privileges":[ "data:write/name ], "resources":[ "abcd" ]} - false

[elasticmachine]

Original comment by @tvernum:

I've updated the description above to reflect a recent change I've made to make the application name explicit.
During testing it became apparent that this would produce a much clearer API.