Implement new extension points in IdentityPlugin and add ContextProvidingPluginSubject

[cwperks]

Description

Companion PR in core: opensearch-project/OpenSearch#14630

This PR by itself does not add additional functionality, it simply implements the new extension points in core and introduces a new class called ContextProvidingPluginSubject which populates a header in the ThreadContext with the canonical class name of the plugin that is executing code using pluginSystemSubject.runAs(() -> { ... }). See ./gradlew integrationTest --tests SystemIndexTests for an example that verifies that a block of code run with pluginSystemSubject.runAs(() -> { ... }) has contextual info in the thread context.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Enhancement

Issues Resolved

Related to #4439

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwperks]

createSecurityRole doesn't align with how these configuration has been used before, its meant to be a representation of the security configuration from the security index, not a way to update/generate/create security objects.

Rather than overload SecurityRole with this responsibility, lets make a new component that is initialized in its own way and separately from our existing configuration. Its strange to have the logic to create these roles inside of configuration model specific code and for its state to be held inside of the privilege evaluator.

That's a good point. I pushed a commit to abstract this out into a separate class called InMemorySecurityRolesV7 and made it distinct from any of the ConfigV7/ConfigV6 classes which are backed by the security index.

[cwperks]

@peternied Here's some mermaid diagrams that I made with the help of ChatGPT-4o

Here's the prompt:

In the plugin installation process:

1. OpenSearch will read both plugin-security.policy (JSM Permissions) + plugin-permissions.yml (requested transport actions) and prompt the administrator about the requested permissions. Administrator can choose to accept the risks or abort the plugin installation process.

In the OpenSearch Bootstrap process:

1. The IdentityService is initialized using a singular IdentityPlugin
2. For each IdentityAwarePlugin, the IdentityPlugin will assign a PluginSubject
3. The plugin uses the pluginSubject to execute transport actions using the identity associated with the plugin

Low-Level Detail

The permissions defined in plugin-permissions.yml are used to create an In Memory role that allows the security plugin to re-use all existing authz logic

pluginSubject.runAs(() -> { ... }) is a direct replacement for try (ThreadContext.StoredContext ctx = threadContext.stashContext()) { ... }

---

Plugin Installation Sequence

sequenceDiagram
    participant Admin as Administrator
    participant OpenSearch as OpenSearch
    participant Plugin as Plugin
    
    OpenSearch ->> Plugin: Reads `plugin-security.policy` (JSM Permissions)
    OpenSearch ->> Plugin: Reads `plugin-permissions.yml` (Transport Actions)
    OpenSearch ->> Admin: Prompt for requested permissions
    
    alt Admin accepts risks
        Admin ->> OpenSearch: Accept permissions
        OpenSearch ->> Plugin: Continue with installation
    else Admin aborts
        Admin ->> OpenSearch: Abort installation
    end

Loading

OpenSearch Bootstrap Sequence:

sequenceDiagram
    participant Bootstrap as OpenSearch Bootstrap
    participant IdentityService as IdentityService
    participant IdentityPlugin as IdentityPlugin
    participant Plugin as IdentityAwarePlugin
    
    Bootstrap ->> IdentityService: Initialize IdentityService
    IdentityService ->> IdentityPlugin: Use singular IdentityPlugin
    
    loop For each IdentityAwarePlugin
        IdentityPlugin ->> Plugin: Assign PluginSubject
        Plugin ->> IdentityPlugin: PluginSubject executes transport actions
    end
    
    Plugin ->> Plugin: Executes actions with associated identity

Loading

[nibix]

Just one heads up that one goal of #4380 is to remove SecurityRoles alltogether.

[peternied]

Thanks for iterating on this @cwperks we are getting closer.

@nibix if you wouldn't mind could you make sure the outstanding comment threads I've created are resolved as appropriate - when this is done please dismiss my review if you think the sprint of the feedback has been addressed. Thanks!

[peternied]

This doesn't seem necessary, if the user principle that represents a plugin is trustworthy the lookup for allowed system indices should return an empty set for a normal user. Lets treat all users the same in code, but allow there configuration data to be populated in a way that distinguishes them well.

By special casing for a plugin user vs normal user create a huge potential for bugs down the line where these permissions are not accounted correctly that would result in a CVE. Note; If there are performance concerns then we need to revisit how these data structures are build to be O(1) lookup.

[cwperks]

That makes sense, I just pushed a change gets rid of the need for the subclass.

I plan to use a username convention for these special "plugin users". The name will be plugin:<pluginCanonicalClassName>. I'm choosing this because : is restricted from usernames, so this will provide assurance that this is a special kind of system username and not a user that could have been created through any other means.

[cwperks]

Sync'ed with the latest from main

[cwperks]

Resolved conflicts after merge of latest PRs

[cwperks]

Just one heads up that one goal of #4380 is to remove SecurityRoles alltogether.

Thank you @nibix! I will check to see what changes are necessary to incorporate this change with Optimized Privileges Evaluation. I'm introducing the "in-memory" role for a plugin in this PR with the goal of eventually incorporating the changes from this PR in core so that we can re-use the authz checks when plugins want to operate outside the authenticated user context.

[peternied]

Thanks for making progress here, I've created some new comment threads and followed up on existing ones.

[peternied]

All User's support runAs(...), not sure what the case for this class is

[cwperks]

Its a vehicle for carrying the subject that's been assigned to a plugin to the transport layer. Take a look at SystemIndexPlugin1 where it instantiates class and returns it from createComponents so that its injectable to transport actions.

[peternied]

Shouldn't special case user vs non-user, seems like they could both be checked against either registry.

[cwperks]

Abstracted this logic to a separate method. The problem with the latter in this block is that its actually a filtering operation and it filters based on the roles in the security index.

For these plugin users, the permissions are not tied to the security index. If opensearch-project/OpenSearch#15778 gets merged, then the permissions given to a plugin are defined in a yaml file that is read on bootstrap and kept in the memory of the process. There is no plan to support dynamic updates to these permissions as they are static and prompted to a cluster administrator on plugin install to accept.

[peternied]

Good idea!

[cwperks]

IMO This should always have been exposed through methods instead of directly accessing fields of this class.

[peternied]

Can we remove the distinction between different kinds of users, it adds complexity in these downstream systems that will make the code harder to maintain over time.

[peternied]

Note; It seems reasonable that no matter the source so long as the expected permissions grants are on the user (if it represents a person or a plugin) they can access system indices even if practically they aren't available at this point in time. What do you think?

[cwperks]

I think this distinction should be kept here since this is an "ownership" model. Authorization is done based on identity here and not a role. i.e. PluginA registers SystemIndexA and gets full access to it by default.

We have a separate mechanism for provisioning system index access to regular user's but it relies on a feature being enabled. The feature flag is plugins.security.system_indices.permission.enabled and its off by default.

[peternied]

Authorization is done based on identity here and not a role. i.e. PluginA registers SystemIndexA and gets full access to it by default.

Why add an alternative authorization model for this case? Seems like this makes the system more complex to understand and verify vs role checks.

[peternied]

@nibix @DarshitChanpura I've got a preference for consistent treatment Users objects no matter if they represent a person or a plugin, I'd be curious for your opinions one way or the other.

[cwperks]

@peternied For what its worth, there is service account specific logic in the same file. Service account is analogous to plugin user, but the difference is that its meant for extensions. i.e. If there are extensions in a cluster that register system indices, then on bootstrap the security plugin issues a token to the extension that it can use to access its registered system indices if it wants to use OpenSearch as a datastore.

Edit: IMO I'd be in favor of unifying the service account logic and plugin user logic longer-term. I think there would need to be some additional changes in the extension registration process to allow configuration of security settings like reserving (system) indices.

[DarshitChanpura]

From what I understand, we are trying to introduce a concept of Plugin Awareness. This requires Identification of the plugin. To enforce this identity we are leveraging already existing User authz flow. Each plugin should have access to its own resource by default, but no other indices. We already have "permission-types" in place, so IMO introducing a plugin user is manageable. We can mark plugin specific users and roles as reserved and static (probly hidden?) to prevent them from being unintentionally modified.

[cwperks]

We can mark plugin specific users and roles as reserved and static (probly hidden?) to prevent them from being unintentionally modified.

These aren't tied to an index. They are purely in-memory and cannot be altered.

[peternied]

Can you help me understand why not? To take this to an extreme, if a user object representing plugin could have bad interactions with existing scenarios, why re-use the User object at all?

[cwperks]

Can you help me understand why not? To take this to an extreme, if a user object representing plugin could have bad interactions with existing scenarios, why re-use the User object at all?

The only real distinction is in that logic to get the "roles". For a regular user, it would filter from the roles that are in the cache, but for a plugin user it would get or create an instance of a role that's only kept in the memory of the process and not backed by the security index. With this PR, pluginSubject's are limited to interactions only with their own system indices. I plan to do a follow-up to this PR once another corresponding PR in core is merged that let's plugins request additional permissions at installation time.

Again, the distinction is in how to compute the role or roles that are used to evaluate permissions. I will abstract the logic into a single method.

[cwperks]

I will open a separate PR rebased on #4380