[Feature/Identity] Prototype Internal IdP

[cwperks]

Description

As part of the identity feature branch, this adds a foundation for building an internal IdP inside of core. This PR includes a working prototype of an IdP similar to the one in the security plugin. I'm creating this as a draft to solicit feedback, there is currently no way to interact with the internal realm via actions in OpenSearch.

This branch shows a custom realm built with shiro called the internal realm and uses BCrypt (salt + hash) for authenticating subjects. I had to add a line in security.policy to get the ObjectMapper working for jackson-databind and also needed to add a dependency on slf4j for shiro, but I'd like to see if there are ways not to include this since opensearch uses log4j for logging.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3653/
• CommitID: 8836831

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3655/
• CommitID: da01d86

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3656/
• CommitID: bc31d46

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3689/
• CommitID: ff50230

[cwperks]

@peternied Do you think we should consider making the work we're doing a native module of OpenSearch?

https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#modules

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3691/
• CommitID: f149788

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3712/
• CommitID: 6387701

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3713/
• CommitID: 15f3aa2

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3714/
• CommitID: 350d7a2

[peternied]

@peternied Do you think we should consider making the work we're doing a native module of OpenSearch?

We spoke about this briefly offline, recapping from the discussion. While we could refactor this to reduce the number of default dependencies on OpenSearch, I think we at the minimum need security to be accessible without installing anything else.

I think this is a good motivation to get a pull request into main sooner so we can see the point for or against it. I might be a good idea to add documentation of any concerns you have in the code so when we do the feature/identity -> main pull request those ideas are surfaced

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3717/
• CommitID: 584b2ff

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3722/
• CommitID: 408dbea

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3724/
• CommitID: 49fa666

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3727/
• CommitID: b58bd58

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/3730/
• CommitID: e96a39f

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/4071/
• CommitID: e29eeea

[cwperks]

@peternied I wanted to leave an update on this branch. I took the work I was doing in the org.opensearch.identity package and created a new library under sandbox called opensearch-authn with the top level package being org.opensearch.authn to match the library name. I think this is the right place to put this work for now until it can be promoted as a lib of opensearch. It also makes it easy to create a build with native authn included or not by specifying ./gradlew assemble -Dsandbox.enabled=false.

One of the problems I faced in this branch is that the build was failing on ./gradlew dependencyLicenses because server got the new transitive dependencies introduced in opensearch-authn. A lot of the deleted files on this PR were no longer needed since the licenses and jars were already in the opensearch-authn library.

There are currently CI issues with jarHell in bouncy castle >.<

class: org.bouncycastle.LICENSE
jar1: /Users/cwperx/.gradle/caches/modules-2/files-2.1/org.bouncycastle/bc-fips/1.0.2.3/da62b32cb72591f5b4d322e6ab0ce7de3247b534/bc-fips-1.0.2.3.jar
jar2: /Users/cwperx/.gradle/caches/modules-2/files-2.1/org.bouncycastle/bcprov-jdk15on/1.70/4636a0d01f74acaf28082fb62b317f1080118371/bcprov-jdk15on-1.70.jar

Bouncy castle is needed by authn for its BCryptHash capabilities which will not be released in Shiro until 2.0.

[peternied]

There are currently CI issues with jarHell in bouncy castle >.<

How would you feel about disabling jarhell? We did this in security because of issues with the Kafka test libraries [1], then we can make an issue and circle back.

It might be as easy as adding the following to server/build.gradle

jarHell.enabled = false

[1] https://github.com/opensearch-project/security/blob/main/build.gradle#L103

[peternied]

Note; I saw that you disabled jar hell in the authz plugin, it looks like the failure is coming out in :distribution:tools:plugin-cli:jarHell is it just that one that is causing issues, otherwise we could unregister the gradle tasks for the time being

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/4167/
• CommitID: 9794416

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/4168/
• CommitID: 47762cb

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/4171/
• CommitID: c6939ce

[cwperks]

There are currently CI issues with jarHell in bouncy castle >.<

How would you feel about disabling jarhell? We did this in security because of issues with the Kafka test libraries [1], then we can make an issue and circle back.

It might be as easy as adding the following to server/build.gradle

jarHell.enabled = false

[1] https://github.com/opensearch-project/security/blob/main/build.gradle#L103

Thank you @peternied. I disabled the jar hell check for plugin-cli and would like to revisit to see if there is another solution to resolve the jar hell problem.

[peternied]

@cwperks Note; I am happy to merge so you can address in follow up PRs

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/4300/
• CommitID: ce56966

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/4336/
• CommitID: fb77c70

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/4357/
• CommitID: e6492ba

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/4363/
• CommitID: 215728b

[cwperks]

@peternied CI is passing again - issues this last time with security policy with jackson-databind and the integration test suite. In a follow-up PR I plan to see if there's another way of transforming yaml -> POJOs to eliminate the dependency on jackson-databind and jackson-annotations for this new library.