feat(kafka): iam roles anywhere + assume role auth profiles

[sicoyle]

Description

This PR:

• Adds the IAM Roles Anywhere auth profile to Kafka
• Moves the disjoint Assume Role fields to their own auth profile. This is only used by a few components, so the description is marked as such. However, it is an auth profile that we can add support for in other components, so I've moved it to the auth profile common AWS area to allow us to add this to additional components in the future.
• Denoted that the originally disjoint Kafka AWS fields become deprecated in Dapr 1.17 as we mv everything to use the common AWS auth profiles that are built in. This meant really gross manual manipulations on the metadata bundle unfortunately...

Metadata bundle changes:
NOTE: This includes all duplicate fields as we deprecate some in the future. Due to this, I've manually made things like accessKey and secretKey that are normally required, not required to not break existing users as we migrate to the common AWS auth fields without the aws prefix.

{
          "title": "AWS: Access Key ID and Secret Access Key",
          "description": "Authenticate using an Access Key ID and Secret Access Key included in the metadata",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsAccessKey",
              "description": "Deprecated as of Dapr 1.17. Use 'accessKey' instead.\nIf both fields are set, then 'accessKey' value will be used.\nAWS access key associated with an IAM account.",
              "type": "string",
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "awsSecretKey",
              "description": "Deprecated as of Dapr 1.17. Use 'secretKey' instead.\nIf both fields are set, then 'secretKey' value will be used.\nThe secret key associated with the access key.",
              "sensitive": true,
              "type": "string",
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "awsSessionToken",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionToken' instead.\nIf both fields are set, then 'sessionToken' value will be used.\nAWS session token to use. A session token is only required if you are using temporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "Deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "awsRegion",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "required": true,
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "accessKey",
              "description": "AWS access key associated with an IAM account",
              "required": true,
              "sensitive": true,
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "secretKey",
              "description": "The secret key associated with the access key",
              "required": true,
              "sensitive": true,
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "example": ""
            }
          ]
        },

Note: this is a new auth profile I've made by moving some of the fields here, so this shows some fields that are becoming deprecated:

        {
          "title": "AWS: Assume specific IAM Role",
          "description": "Assume a specific IAM role. Note: This is only supported on Kafka and PostgreSQL components.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsIamRoleArn",
              "description": "Deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "assumeRoleArn",
              "description": "IAM role that has access to AWS resource.\nThis is another option to authenticate with MSK and RDS Aurora aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "sessionName",
              "description": "The session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            }
          ]
        },

Note: This remains as is with the manual manipulation to remove the deprecated fields that are unused for this profile:

        {
          "title": "AWS: Credentials from Environment Variables",
          "description": "Use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            }
          ]
        },

Note: new profile added that I manually remove all the fields that we are deprecating bc they are not needed for this auth profile

        {
          "title": "AWS: IAM Roles Anywhere",
          "description": "Use X.509 certificates to establish trust between AWS and your AWS account and the Dapr cluster using AWS IAM Roles Anywhere.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "trustAnchorArn",
              "description": "ARN of the AWS Trust Anchor in the AWS account granting trust to the Dapr Certificate Authority.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:trust-anchor/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "trustProfileArn",
              "description": "ARN of the AWS IAM Profile in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:profile/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "assumeRoleArn",
              "description": "ARN of the AWS IAM role to assume in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:iam:012345678910:role/exampleIAMRoleName"
            }
          ]
        }

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

[sicoyle]

/ok-to-test

[sicoyle]

unit tests are failing due to wasm issues that are unrelated

[dapr-bot]

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: a1f1c9a

[dapr-bot]

Components conformance test

🔗 Link to Action run

Commit ref: a1f1c9a

❌ Some conformance tests failed

These tests failed:

• bindings.azure.eventgrid

[dapr-bot]

Components certification test

🔗 Link to Action run

Commit ref: a1f1c9a

❌ Some certification tests failed

These tests failed:

• pubsub.aws.snssqs
• pubsub.gcp.pubsub
• state.azure.cosmosdb
• state.sqlserver

[famarting]

overall lgtm

[sicoyle]

/ok-to-test

[dapr-bot]

Components certification test

🔗 Link to Action run

Commit ref: 16834ed

❌ Some certification tests failed

These tests failed:

• pubsub.aws.snssqs
• pubsub.gcp.pubsub
• state.azure.cosmosdb
• state.sqlserver

[dapr-bot]

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: 16834ed

[dapr-bot]

Components conformance test

🔗 Link to Action run

Commit ref: 16834ed

❌ Some conformance tests failed

These tests failed:

• bindings.azure.eventgrid
• state.azure.cosmosdb
• state.sqlserver

[sicoyle]

final metadata updates

{
          "title": "AWS: Access Key ID and Secret Access Key",
          "description": "Authenticate using an Access Key ID and Secret Access Key included in the metadata",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsRegion",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'region' instead.\nThe AWS Region where the AWS Relational Database Service is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "awsAccessKey",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'accessKey' instead.\nIf both fields are set, then 'accessKey' value will be used.\nAWS access key associated with an IAM account.",
              "type": "string",
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "awsSecretKey",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'secretKey' instead.\nIf both fields are set, then 'secretKey' value will be used.\nThe secret key associated with the access key.",
              "sensitive": true,
              "type": "string",
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "awsSessionToken",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionToken' instead.\nIf both fields are set, then 'sessionToken' value will be used.\nAWS session token to use. A session token is only required if you are using temporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "region",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "accessKey",
              "description": "AWS access key associated with an IAM account",
              "sensitive": true,
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "secretKey",
              "description": "The secret key associated with the access key",
              "sensitive": true,
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "example": ""
            }
          ]
        },
        {
          "title": "AWS: Assume specific IAM Role",
          "description": "Assume a specific IAM role. Note: This is only supported on Kafka and PostgreSQL components.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsRegion",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'region' instead.\nThe AWS Region where the AWS Relational Database Service is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "region",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "assumeRoleArn",
              "description": "IAM role that has access to AWS resource.\nThis is another option to authenticate with MSK and RDS Aurora aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "sessionName",
              "description": "The session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            }
          ]
        },
        {
          "title": "AWS: Credentials from Environment Variables",
          "description": "Use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            }
          ]
        },
        {
          "title": "AWS: IAM Roles Anywhere",
          "description": "Use X.509 certificates to establish trust between AWS and your AWS account and the Dapr cluster using AWS IAM Roles Anywhere.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "trustAnchorArn",
              "description": "ARN of the AWS Trust Anchor in the AWS account granting trust to the Dapr Certificate Authority.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:trust-anchor/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "trustProfileArn",
              "description": "ARN of the AWS IAM Profile in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:profile/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "assumeRoleArn",
              "description": "ARN of the AWS IAM role to assume in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:iam:012345678910:role/exampleIAMRoleName"
            }
          ]
        }

[sicoyle]

/ok-to-test

[dapr-bot]

Components certification test

🔗 Link to Action run

Commit ref: d3d5953

❌ Some certification tests failed

These tests failed:

• bindings.azure.blobstorage
• bindings.azure.eventhubs
• pubsub.aws.snssqs
• pubsub.gcp.pubsub
• state.azure.cosmosdb
• state.sqlserver

[dapr-bot]

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: d3d5953

[dapr-bot]

Components conformance test

🔗 Link to Action run

Commit ref: d3d5953

❌ Some conformance tests failed

These tests failed:

• bindings.azure.eventgrid
• state.azure.cosmosdb

[sicoyle]

/ok-to-test

[dapr-bot]

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: d3d5953

[dapr-bot]

Components conformance test

🔗 Link to Action run

Commit ref: d3d5953

❌ Some conformance tests failed

These tests failed:

• bindings.azure.eventgrid
• bindings.azure.eventhubs
• state.azure.cosmosdb

[dapr-bot]

Components certification test

🔗 Link to Action run

Commit ref: d3d5953

❌ Some certification tests failed

These tests failed:

• bindings.azure.servicebusqueues
• pubsub.gcp.pubsub
• state.azure.cosmosdb
• state.memcached
• state.sqlserver

[sicoyle]

/ok-to-test

[dapr-bot]

Components conformance test

🔗 Link to Action run

Commit ref: 0ce8e18

❌ Some conformance tests failed

These tests failed:

• bindings.azure.eventgrid
• state.azure.cosmosdb
• state.sqlserver

[dapr-bot]

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: 0ce8e18

[dapr-bot]

Components certification test

🔗 Link to Action run

Commit ref: 0ce8e18

❌ Some certification tests failed

These tests failed:

• bindings.azure.eventhubs
• pubsub.gcp.pubsub
• state.azure.cosmosdb
• state.memcached

[sicoyle]

this pr is ready pls. All of the certification/conformance test failures are unrelated.