Connector middleware

[sagikazarmark]

Right now connectors implement certain types of behavior that could be reused. The most common example is filtering groups.

Consider creating a common middleware layer for connectors instead of implementing these features (and configuration for them) separately for each connector.

[srenatus]

The most common example is filtering groups.

💯 It would be great to refactor that at least. If it fits a more generic pattern (as you described) all the better for it. 👍

[blaggacao]

related → #1790

[merlindorin]

+1 for this kind of pattern.

Maybe there is something to do/learn from ory oathkeeper: https://www.ory.sh/oathkeeper/docs/pipeline/mutator

(by the way... this kind of feature is awesome and we dream about it @renault :) : https://www.ory.sh/oathkeeper/docs/pipeline/mutator#hydrator)

[al45tair]

We're going to want something like this too. If anyone has any suggestions about what designs they might prefer, I might have some time to work on this.

[al45tair]

OK, so my assumption here is that the middleware layer will be passed the identity returned by a successful invocation of a Connector, and will return an (Identity, error) pair. Additionally, multiple middleware components can be stacked, in which case the identity output of one will be passed into the identity input of the next in the stack.

In addition, it looks like the Identity struct will need to be extended to enable custom claims, since middleware that adds custom claims is something people are likely to be interested in (in addition to being able to mutate group names and things). I know we are.

[merlindorin]

I'm fully agree with that.

As a use case : we operate an homemade extensions on Keycloak 😔 in order to retrieve custom claims in order to enrich the user identity. This extension is a simple http call. I easily imagine a Dex "connector" that can do the same.

This kind of feature is for us very precious.

[janwillies]

It would be very interesting to issue an HTTP request from dex and let OPA do the authorization for which claims to include, maybe even handover additional infos like HTTP request headers

[srenatus]

💭 Just sharing a thought here: It could end up nicely if Dex embedded OPA, exposing an interface similar to what gatekeeper does.

A sketchy adaptation of that model to Dex would look like this:

1. There are contraints, like

constraints:
  - kind: allowedGroups
    match:
    - connector: ldap # some connector ID
    parameters:
      groups:
       - ops
       - admin

2. and there's constraint templates, like

spec:
  crd:
    spec:
      names:
        kind: allowedGroups
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            groups:
              type: array
              items: string
  targets:
    - rego: |
        package allowedgroups

        violation[{"msg": msg, "details": {"existing_groups": existing}}] {
          existing := {group | input.review.identity.groups[_]}
          allowed := {group | group := input.parameters.groups[_]}
          overlap := allowed | provided # set intersection
          count(overlap) == 0
          msg := sprintf("must be in one of the allowed groups: %v", [allowed])
        }

The most common constraint templates would be shipped with Dex, but it would also allow creating your own ones, and using them. The constraints would live in the config file, the constraint templates maybe in some other place.

The sketch doesn't include how you'd have the constraints eventually alter the groups, or other claims, of a user, but that's something that could surely be done; but may have to be discussed, too.

This scheme has proven useful for gatekeeper, I believe, as it allows most usecases to be done from configuration without messing with Rego and OPA, but leaves an escape hatch for everyone who has special needs.

Furthermore, Rego policies can do HTTP requests, so this could be an extension point for using another system.

More information about the constraint framework can be found in https://github.com/open-policy-agent/frameworks/tree/master/constraint.

[sagikazarmark]

That sounds really interesting.

As far as I can tell, this is mostly useful for policy enforcement. As you also pointed out, it could alter groups and other claims, but I wonder if we should take a more generic approach and make OPA/rego an implementation detail providing policy enforcement.

[srenatus]

@sagikazarmark Yeah it would also make for a decent middleware piece, sure. 😃

[candlerb]

I note that this PR adds a CustomClaims to connector.Identity.

I think that changing connector.Identity in this way is a pre-requisite for external connectors (#1907) as well. If the connectors can't provide custom claims from upstream IDPs, then there will be less information for the middleware to work with.

[sagikazarmark]

I'd like to pick this up after my PTO. I'd like to prepare a POC implementation, supporting static connectors for now.

I also wonder if middleware should be available for clients as well. (eg. filter groups for a specific client so that login can be denied/limited to certain groups)

[nabokihms]

The idea of something related to the topic came to my mind.

What if after a new id token is generated for the user, Dex can validate/mutate it by sending a request to some webhook (like Kubernetes does) or by calling a script (like docker registry does)? With this approach we will be able to provide all required customization for users and keep it simple for Dex.

[sagikazarmark]

I think it would still make sense to bundle a bunch of middleware with Dex (like Kubernetes does), but extensibility is certainly important as well.

[benjamin-bergia]

Would this allow modifying groups returned by the connectors or maintain some sort of mapping?
I currently have an instance with multiple github connectors to different organizations. Not all organizations use the same naming conventions for their teams and I would like to have a mapping on Dex to rename groups with something that makes more sense internally. For example my gh connectors might return groups like orgA:devs, MySuperOrg:development and I would like Dex to rewrite both into orga-dev, mysuperorg-dev.

[sagikazarmark]

Yep, that's absolutely one of the use cases we would like to support.

[robertkaelin]

Hi Everyone
Is this initiative still ongoing or not?

Not having this functionality is what is currently keeping us from using Dex and we are between forking this project or contributing, but there I saw a lot of PR related to this topic being blocked/denied due to uncertainties on how to implement.

Is there anything we could help/build on to provide this, what does still need to be done?

[candlerb]

You could see if Hashicorp Vault would meet your needs: it gained OIDC Provider functionality in 1.9+. It can directly map individual users (entities) and groups to upstream OIDC subscribers and group claims, via aliases. For me that removes the need for middleware.

And there's always Keycloak.

[nabokihms]

@robertkaelin, we are open to any help.

The current step to implement the feature, I believe, is to write a design doc - Dex Enhancement Proposal. There are many ways to solve the problem by utilizing various approaches, technologies, and libraries.

[sagikazarmark]

I plan to work on this a bit in the upcoming weeks, although I'm not completely sure we are at a state to formalize a proposal.

A couple thoughts:

Problem we are trying to solve: users want to inject custom logic at certain layers. We can hardly add support for all of them in Dex, because they are highly user-specific and makes the code unmaintainable.

So far this has been mostly requested on the connector level, but some features (eg. allow/block listing) may be useful on the client level as well.

Challenges:

• Types of middleware
  • Mutations (add/edit/delete values/claims)
  • Gates (allow/block lists)
• Configuration
  • Need to support generic configuration for different kinds of middleware
  • Support configuration for non-statically configured connectors/clients (DB schema??)
• Extension
  • Extend Dex without recompiling
  • Go plugin?
  • Hashicorp go-plugin?
  • WASI?
• Context
  • Current features operate directly on an identity
  • We should be able to access connector specific details (eg. connector specific claims)
• Testing
  • How do we test middleware?

[nabokihms]

I think we can go just with the connector middleware level and extend it for clients in the future. Speaking about extensions, I still lean to use a language like rego or CEL (sufficient for mutation and validation, easy to test).

[candlerb]

<bikeshed>jsonnet</bikeshed>

[brianv0]

For Mutation, can you just add some gRPC support in finalizeLogin or something?

[blairdrummond]

I think we can go just with the connector middleware level and extend it for clients in the future. Speaking about extensions, I still lean to use a language like rego or CEL (sufficient for mutation and validation, easy to test).

Just as an anecdote, building on-top of ExtendPayload interface support and the recent Token-Exchange feature, I (badly) added Lua support to a fork of Dex for a proof of concept. The usage looks like:

# config.yaml
connectors:
  - config:
      issuer: https://token.actions.githubusercontent.com
      scopes:
        - openid
        - groups
      userNameKey: sub
    id: github-actions
    name: github-actions
    type: oidc
    tokenMapper.lua: |
        run = string.format("%s-%s-%s", subject.run_id ,  subject.run_number ,  subject.run_attempt)
        principal_tags = {
          repo             = { subject.repository },
          sha              = { subject.sha },
          actor            = { subject.actor },
          env              = { subject.environment },
          event            = { subject.event_name },
          ref              = { subject.ref },
          ref_type         = { subject.ref_type },
          job_workflow_ref = { subject.job_workflow_ref },
          visibility       = { subject.repository_visibility },
          runner           = { subject.runner_environment },
          run              = { run }
        }

        local keyset = {}
        for k,v in pairs(principal_tags) do
        keyset[#keyset + 1] = k
        end

        token["https://aws.amazon.com/tags"] = {
          principal_tags = principal_tags,
          transitive_tags = keyset
        }

The end result of this is I can basically implement what Aidan Steele described in his blog post AWS IAM OIDC IDPs need more controls, i.e. I can authorize assume role in AWS using any attributes in the underlying JWT by mapping them to principal tags. Imho this is an enormously useful use-case.

[blairdrummond]

Follow-up, I did it again in Rego, too.

liatrio@0957553

  - config:
      issuer: https://token.actions.githubusercontent.com
      scopes:
        - openid
        - groups
      policy.rego: |+
          package token
          import rego.v1
          result := {
            "allow": allow,
            "token": token
          }

          # don't let all of github token-exchange against dex
          allow := input.claims.repository_owner == "liatrio"

          default token := {}

          token := x if {
            x := object.union(input.claims, input.token)
          }
      userNameKey: sub
    id: github-actions
    name: github-actions
    type: oidc

[dee-kryvenko]

Multiple people did multiple PRs, multiple forks etc and this is still not moved for over 4 years now. Never thought I would see a day where things would be moving faster and more nimble at my giant transnational employer than in the open source world. Sadly it is a new norm. Everyone want to play with big toys now, run Enhancement Proposals, boards, committees, do virtually anything else other than spend time writing code. So sad.