Improvements to the namespace Role Mappings

[costrouc]

Motivation

For authentication currently the user to role and permissions has been managed in traitlets. Via a few important attributes

c.RBACAuthorizationBackend.role_mappings: Dict[str, set[str]] = {
            "viewer": {
                schema.Permissions.ENVIRONMENT_READ,
                schema.Permissions.NAMESPACE_READ,
             }
 }       
c.RBACAuthorizationBackend.unauthenticated_role_bindings: Dict[str, set[str]] =  {
        "default/*": {"viewer"},
 }
c.RBACAuthorizationBackend.authenticated_role_bindings: Dict[str, set[str]] = {
            "default/*": {"viewer"},
            "filesystem/*": {"viewer"},
}

class MyAuthentication(Authentication):
       async def authenticate(self, request: Request):
            return schema.AuthenticationToken(
                primary_namespace=username,
                role_bindings={
                    "*/*": ["admin"],
                },
            )

c.CondaStoreServer.authentication_class = MyAuthentication

Developers were responsible for creating a function authenticate(...) which handled the user to role mapping. With this configuration it was possible to conda-store to not know anything about the users.

Proposal

I propose adding one new database table to conda-store. We leverage the fact that the username will match a namespace name which is created on the backend.

# orm.py

# update namespace table
class Namespace(Base):
     ...
     metadata = Column(JSON) # would be nice to rely on metadata so that we don't enforce the attributes that are on a user

class NamespaceRoleMapping(Base):
    id = Column(Integer, primary_key=True)
    namespace_id = ...
    entity = Column(Unicode(255), nullable=False)     # arn e.g. <namespace>/<name> like `quansight-*/*` or `quansight-devops/*`
    role = Column(Unicode(255), nullable=False)  # e.g. viewer

Then add a method to RBACAuthorizationBackend

def get_namespace_role_bindings(namespace: str):
      # query database for all entity role mappings
      return {entity: roles} # will match form in `schema.AuthenticationToken.role_bindings`

Additionally there should be rest api methods and permissions for CRUD. I am least sure about this part.
- need to add a schema.Permissions namespace::update permission which will allow updating the metadata
- namespace update rest api method that uses the namespace update
- create, list, delete for adding namespace role mappings. I think you should only be able to edit namespace role mappings if you have namespace::update permissions. If you have namespace::read it should be sufficient to view members.

@pierrotsmnrd I will be gone until Thursday but I think this should be scoped well enough to start. I will periodically be checking github so I should be able to respond to comments.

[pierrotsmnrd]

First step with the DB impacts is here in branch 491-role-mapping

Then I'm kind of unsure how to progress with get_namespace_role_bindings. Whats the best way to get access to the DB from this class that doesn't have access to it ?

[costrouc]

I'd just supply the db session as a function argument. Would that work?

[pierrotsmnrd]

Ok, some parts around the authentication were still blurry but I see clearer now.

I have explored the solution you have described. Let me know if I get this correctly, because I have a slightly different approach to suggest:

1. My understanding is that all the filter_* functions in class Authentication ( Authentication::filter_builds, Authentication::filter_environments, Authentication::filter_namespaces ), will need to rely on get_namespace_role_bindings, directly or indirectly, to filter the results from the API according to the role bindings.

2. I noticed that all these functions receive an entity parameter, which is an AuthenticationToken that contains a role_bindings property.

3. So couldn't we rely on entity.role_bindings instead ? The property would have to be set up when the AuthenticationToken is built. This approach would have less impacts in the codebase (as far as I understand it, but I might be missing something that prevents this approach). An extra feature would be required : updating the role bindings regularly based on the DB.

Let me know what you think :)

[pierrotsmnrd]

2023 07 06 - Sync with Chris :

• RBACAuthorizationBackend needs a new method like database_role_bindings that will return a structure similar to authenticated_role_bindings and unauthenticated_role_bindings
• inject that function in get_entity_bindings - fix the merge of the sets (lines 159-168)
• Add some checks in authorize_request / authenticate_request, might require to add a DB handle to the authentication object

[costrouc]

Great summary @pierrotsmnrd. After checking the code more I think that several methods will require the db argument.

• auth.authorization.get_entity_bindings
• auth.authorization.get_entity_binding_permissions
• auth.filter_namespaces
• auth.authorize_request
• api.list_environments

The more I think about it the RBACAuthorization class just generally needs a db session object injected into it when being created. . It would be difficult to pass the db instance in all methods.

Might be easiest to add after https://github.com/Quansight/conda-store/blob/main/conda-store-server/conda_store_server/server/app.py#L189 the following:

self.authentication._authorization.db = conda_store.db

Probably do in a more official way 😄

[costrouc]

This issue is partially implmented @pierrotsmnrd. Now we need to add api methods mentioned in this issue.

[costrouc]

I think this issue is fully resolved now that we have merged the PR #508. Will re-open if issues occur.

[costrouc]

Reopening since we need to make get namespace return the api endpoint data.

[costrouc]

This is an important issue for the role mappings in the database to be done properly but is not a priority for JATIC work.

[trallard]

@nkaretnikov assigning this to you

[costrouc]

This is a complex issue that I think is worth talking about over a call with @pierrotsmnrd and I. I'll put the current status here.

I think we slightly missed how this should be implemented.

The entire use case:

I chris would have namespace-role-mapping::create and namespace-role-mapping::delete permissions on a particular namespace say quansight. This means I should have enough permissions to "grant" user pierre a role on the quansight.

We have a database table orm.NamespaceRoleMapping which is used to track these permissions. I think the entity field should be instead be something like target_namespace_id. Meaning user namespace_id has role role on specific namespace target_namespace_id. I don't think that entity helps us here and @pierrotsmnrd sorry for the wrong direction here.

Also I have doubts on us having PUT /namespace/{namespace}/ as the route for updating role_mappings. The more I think about it I think I'd prefer something like an explicit domain e.g. PUT /namespace/{namespace}/role_mappings/.

So I think that most of the hard work is already done but we need to tweak the REST api and fixup how we represent things in the DB. @pierrotsmnrd do you have thoughts? Also we should schedule a call.

[nkaretnikov]

Status update for PR #607:

• the backend/tests are functionally ready
• got an initial LGTM from Chris on the PR
• Tania asked for the UI to be done by Smera, who should be able to look into that in ~1 week IIUC
• TODO: the PR needs to be updated to comply with the backward compat policy
• I also need to publish the said BC policy (it's been discussed, I need to transfer it from my notes)
• the to-do items above will be done this weekend.

[costrouc]

@nkaretnikov has opened a draft PR and needs my review.

[nkaretnikov]

@smeragoel and I had a call about the design part of this. I also talked to @costrouc after that.

• We'll need to create a completely new page that should be accessible from the homepage (new UI) to edit namespace permissions (not part of PR 607 since it's UI)
• Note: sharing is on the namespace level. When you give a role to someone, that person can access all environments in this particular namespace. If you need different permissions, then create multiple namespaces. This is not as flexible as per-environment permissions, but easier to implement and design UI for
  • Per-environment sharing can be added later
  • This also supports the most common use-case that people have (I want to share all these environments between team members)
  • To make this user-friendly, we'll need to:
    • Make it possible to move environments between namespaces in the UI
    • Allow creating new namespaces in the new UI (this already exists in the admin interface in http://localhost:8080/conda-store/admin/namespace/)
• Conceptually we need (and have) these things (all routes are part of PR 607, prefixed with /api/v2, see conda-store-server/conda_store_server/server/views/api.py):
  • Sharing a namespace (give someone access to this namespace)
    • post("/namespace/{namespace}/role" - give namespace access to this namespace with some role (viewer, developer, admin)
      • parameters: current namespace in URL, other namespace name, role (viewer, developer, admin)
  • Viewing shared permissions (who can access this namespace?)
    • get("/namespace/{namespace}/roles" - view all users/namespaces who have access this namespace
      • parameters: current namespace in URL
    • get("/namespace/{namespace}/role" - same but for one user
      • parameters: current namespace in URL, other namespace name
  • Modifying those permissions
    • upgrading/downgrading user permissions
      • put("/namespace/{namespace}/role" - change access role of namespace (to viewer, developer, admin)
        • parameters: current namespace in URL, other namespace name, role
    • removing users
      • delete("/namespace/{namespace}/roles" - delete all namespaces who had access to this namespace
        • parameters: current namespace URL
      • delete("/namespace/{namespace}/role" - same but for one user
        • parameters: current namespace in URL, other namespace name.

[nkaretnikov]

Status update:

• I'll get back to this PR ASAP once I have hours to work on this. Maybe this week if I find time during my unallocated hours.
• We've agreed on BC/FC policy, so I need to make those changes.
• Then I'll ping Chris or Chuck to review so this goes forward.
• Once this is merged, I'll do a quick proof of concept to check that we have all the bits to make this practical for users. As indicated above, I might need to add a way to move environments between namespaces, since role mappings are currently namespace-based.
• @smeragoel could you share the status of the design part of this? (Not a blocker for my server PR, but we need to have a design to implement and ship the UI component to users.)

[smeragoel]

I’ll get a first draft if the designs ready for review by the end of this week!

[smeragoel]

This is a first draft of the designs required for the workflow described above.

1. Namespace Page

1. This is a completely new page that contains details about a particular namespace.
2. It is accessed by clicking on the namespace name (Kim Pevey's Namespace) on the left panel. This introduces a new behaviour. Right now, clicking on Namespace Name expands the dropdown on the left panel to show all the environments. According to the proposed design, clicking on it will expand the dropdown and show the namespace page also.
3. Right now, this screen only contains information about the contained environments and user access, but we can also add other metadata also.

---

2. Edit Namespace

1. This is where you would adjust permissions for the namespace.

---

3. Creating a new namespace

1. I am having some trouble with this. While the flow is pretty straightforward, I am unsure of the positioning of the Create New Namespace button. Here are some design considerations for the button:

   • It should be in the left panel since it is always on screen and the user can access it from any screen.
   • The button should say Namespace to avoid confusion with the New Environment button.
   • The button should ideally be above Shared Namespaces heading to avoid signalling that you can only created shared namespaces.

2. With these in mind, I positioned the button as such. I am not 100% happy with the placement, but I think having some fresh eyes on it would help.

3. This is how the create new namespace page would look like, after clicking on the button above. This has very basic fields right now, name and user access, but it can also contain other properties/metadata.

[kcpevey]

@smeragoel these look great! Thank you!

New Namespace button - I think the placement looks good. The label is kind of long for a button, but I agree with you that its necessary to include the word "namespace" so I dont think anything can be done about it.

What is the checkbox next to the trashcan for? - Ohhh now I see that line is what it looks like if you change the role. On the New Namespace page, the changing role highlighting and checkmark doesn't make sense though, right? Since everything will be new?

One thing we might have to think through is adding Users to the new Namespace.

• Can I add a group of people from another Namespace/keycloak group to a new namespace? Usecase: I'm creating an environment for two teams to use. Do I have to add each of their names manually or can I just add those two teams keycloak user groups?
• as an end-user with no elevated priveledges how will I find user names? Does this field contain autocomplete or a dropdown selection?

These questions are mixed backend and frontend and I realize they may initiate more work so I want to say that I'm happen to consider these ideas as enhancements once we get the POC out.

[pavithraes]

From meeting today, it'll be intuitive to rename the "developer" role to "editor"

[nkaretnikov]

From meeting today, it'll be intuitive to rename the "developer" role to "editor"

After making this change locally, I realized it introduces too many changes. It requires updating the default role mappings, the docs, the DB validation functions, and the tests (to support both of these roles). The current role mappings PR is already too big. This will be done separately. New issue: #675.

[nkaretnikov]

From the meeting today:

• Research if there's a way to transparently migrate the old role mappings table to the new one in the DB (then v2 in CLI can be the default). Do not spend more than a few hours on this.
• If not, document it and the implications for users (mention that DB tables are separate)
• Chuck also suggested a separate script that might do a partial migration and will tell users which entries were not migrated. This is a separate task, which we'll prioritize according to workload, and can also be done later.

[nkaretnikov]

Migrating data from v1 to v2 role mappings will be done in a separate PR because it can cause issues during migration, which might require manual intervention. So a standalone function will be provided for those who need it. #681

[nkaretnikov]

Status update: asked Chris and Chuck to review PR #607. Also, allocated time for a meeting to answer questions to speed up review process.