Add AWS CloudFormation Provisioning and SDK Configuration

[normj]

I’m looking for feedback on this approach that will help support provisioning AWS application resources and also configure service clients from the AWS SDK for .NET. I view these features as the minimum level to build higher components or integrations like CDK and deployments.

For .NET developer that use AWS, please let me know via 👍 or comments if you find this useful. Your feedback will help us understand if there is enough interest from our customers to prioritize more AWS Aspire work.

CloudFormation was chosen so that new AWS features will be available to Aspire components as they are released by AWS. This eliminates the need for Aspire components to be updated which could create a blocker for some customers. Granted sometimes there is a delay for CloudFormation support but that delay has gotten pretty small. Also using CloudFormation makes easy to tear all of the applications resources as a single unit.

SDK Service Client Config

To configure service clients you would define an IAWSSDKConfig and using fluent programming flow add the properties for service client. Currently the only thing supported is credential profile and region.

var awsConfig = builder.AddAWSSDKConfig("frontend-aws-config")
                        .WithProfile("default")
                        .WithRegion(RegionEndpoint.USEast2);

The WithReference extension method for IResourceBuilder<ProjectResource> is used to attach the SDK config to the project.

builder.AddProject<Projects.Frontend>("frontend")
        .WithReference(awsConfig);

During the startup of AppHost environment variables are applied to the project where AWSSDK.Extensions.NETCore.Setup used in the project will look for via IConfiguration.

builder.Services.AddAWSService<IAmazonSQS>();
builder.Services.AddAWSService<IAmazonSimpleNotificationService>();

This approach allows multiple configurations to be defined in the AppHost and assign them to the appropriate projects. I suspect it is common users will want a default/global IAWSSDKConfig that should be tied to all projects automatically unless the project was given a specific IAWSSDKConfig. I haven't seen a way for an object like IAWSSDKConfig to be able to iterate through all other projects and apply environment variables. If there is a way I would appreciate hearing how.

Provisioning

Provisioning is done by adding a CloudFormation template to the AppHost, either JSON or YAML. As part of the IDistributedApplicationBuilder in Program.cs you would add the CloudFormation provisioning using the AddAWSCloudFormationTemplate. The first parameter is the name of the CloudFormation stack and the second is the CloudFormation template file used to create the stack.

var awsResources = builder.AddAWSCloudFormationTemplate("AspireSampleDevResources", "app-resources.template")
                        // Add the SDK configuration so the AppHost knows what account/region to provision the resources.
                        .WithReference(awsConfig);

During AppHost startup a IDistributedApplicationLifecycleHook is used to check if the stack exists and if not create the stack and wait till the stack is complete which means the AWS resources are available for projects to use. Depending on the types and number of resources defined in the CloudFormation template this can take some time to allocation.

To keep the F5 dev cycle quick a tag is assigned to the CloudFormation stack with a SHA256 of the CloudFormation template. During subsequent starts of the AppHost the SHA256 of the CloudFormation template is compared and if it is the same as the value in the tag it skips CloudFormation updates.

In a possible deployment story in the future I could see the CloudFormation template of applications resources be a nested stack in a deployment CloudFormation stack.

Associated AWS resources to a project

CloudFormation templates have output parameters that are often used to return the identifiers of resources to use in code. For example this CloudFormation template puts the identifiers for a queue and a topic as output parameters.

{
    "AWSTemplateFormatVersion" : "2010-09-09",
    "Resources" : {
        "ChatMessagesQueue" : {
            "Type" : "AWS::SQS::Queue",
            "Properties" : {
            }
        },
        "ChatTopic" : {
            "Type" : "AWS::SNS::Topic",
            "Properties" : {
                "Subscription" : [
                    {"Protocol" : "sqs", "Endpoint" : {"Fn::GetAtt" : [ "ChatMessagesQueue", "Arn"]}}
                ]
            }
        }
    },
    "Outputs" : {
        "ChatMessagesQueueUrl" : {
            "Value" : { "Ref" : "ChatMessagesQueue" }
        },
        "ChatTopicArn" : {
            "Value" : { "Ref" : "ChatTopic" }
        }
    }
}

A CloudFormation stack can be associated to project using the WithReference extension method.

builder.AddProject<Projects.Frontend>("frontend")
        .WithReference(awsResources)

This takes all of the output parameters from the stack and assigns them as environment variables that applications can pick up through IConfiguration. The default section in IConfiguration that WithReference puts the output parameters in is AWS:Resources but can be overridden with the WithReference method.

Alternatively a single output parameter can be bound to a project via an environment variable.

builder.AddProject<Projects.Frontend>("frontend")
        .WithEnvironment("ChatTopicArnEnv", awsResources.GetOutput("ChatTopicArn"))

Referencing existing stacks

The AddAWSCloudFormationStack is used to include the output parameters of an existing stack. This allows users to handle provision their resources outside of their AppHost but bind the outputs to projects using the same WithReference and WithEnvironment methods. This is particular useful when developers do not have write permissions to CloudFormation.

var awsResources = builder.AddAWSCloudFormationStack("AspireSampleDevResources")
                        .WithReference(awsConfig);

Sample

To demonstrate how these hosting components work and give me a playground to try things out I added a new Aspire AWS sample application. I'm a terrible frontend developer but the frontend has a page that I use basically as diagnostics to see the AppHost configuration filter through AppHost to the application.

The Message Publisher page is a sanity check that the service clients are working with the configs passed in.

AppHost

Here is the full code of the Program.cs for the AppHost using AWS to get an idea how you would use it.

using Amazon;

var builder = DistributedApplication.CreateBuilder(args);

// Setup a configuration for the AWS .NET SDK.
var awsConfig = builder.AddAWSSDKConfig()
                        .WithProfile("default")
                        .WithRegion(RegionEndpoint.USWest2);

// Provision application level resources like SQS queues and SNS topics defined in the CloudFormation template file app-resources.template.
var awsResources = builder.AddAWSCloudFormationTemplate("AspireSampleDevResources", "app-resources.template")
                        .AddParameter("DefaultVisibilityTimeout", "30")
                        // Add the SDK configuration so the AppHost knows what account/region to provision the resources.
                        .WithReference(awsConfig);

// To add outputs of a CloudFormation stack that was created outside of AppHost use the AddAWSCloudFormationStack method.
// then attach the CloudFormation resource to a project using the WithReference method.
//var awsExistingResource = builder.AddAWSCloudFormationStack("ExistingStackName")
//                        .WithReference(awsConfig);

builder.AddProject<Projects.Frontend>("Frontend")
        // Demonstrating binding all of the output variables to a section in IConfiguration. By default they are bound to the AWS::Resources prefix.
        // The prefix is configurable by the optional configSection parameter.
        .WithReference(awsResources)
        // Demonstrating binding a single output variable to environment variable in the project.
        .WithEnvironment("ChatTopicArnEnv", awsResources.GetOutput("ChatTopicArn"))
        // Assign the SDK config to the project. The service clients created in the project relying on environment config
        // will pick up these configuration.
        .WithReference(awsConfig);

builder.Build().Run();

Example Startup

This animated gif gives a high level view of the experience this AWS component in the Aspire Dashboard.

Microsoft Reviewers: Open in CodeFlow

[normj]

It would be great to get these packages added to the Aspire NuGet feeds. Some of these are newer versions of packages that are already in there. Currently I'm testing with hacking the NuGet.config to allow the public NuGet feed.

[davidfowl]

Are these the versions you need?

[eerhardt]

Currently I'm testing with hacking the NuGet.config to allow the public NuGet feed.

Would it be possible to use the "hack" while the PR was in draft, and adding more dependencies? (and possibly updating new versions that are released). Then once the PR is ready for review, we can revert the nuget.config change, and mirror the necessary packages to the trusted feed.

[normj]

Adding the AWS .NET SDK Otel provider. Wonder how a component could possible add it without access to the TracerProviderBuilder.

[davidfowl]

Adding the AWS .NET SDK Otel provider. Wonder how a component could possible add it without access to the TracerProviderBuilder.

Multiple calls to AddOpenTelemetry().WithTracing(t => ...) compose. This is what our aspire components do.

[davidfowl]

I'm choosing the CloudFormation path with a template because I want to avoid any solutions that requires Aspire components to have constantly be updated as AWS adds new features. Using CloudFormation any new features AWS releases will be available to Aspire via CloudFormation without anybody having to wait till somebody updates something. Granted sometimes there is a delay for CloudFormation support but that delay has gotten pretty small. Also using CloudFormation makes easy to tear all of the applications resources as a single unit.

What about a CDK based approach? That might gel better with the code focused model aspire has.

[Kralizek]

What about for those who use Terraform or other IaC tooling?

[normj]

What about a CDK based approach? That might gel better with the code focused model aspire has.

I don't view this as either CloudFormation or CDK. CloudFormation is the lowest level and is very common to use. So I wanted to tackle that first. I suspect we should also add CDK support as well but that would be a separate PR in my opinion. What we can learn here though is if the association of CloudFormation output parameter to projects works for users. CDK at the end of day is still CloudFormation so the association mechanism would be the same.

[davidfowl]

What about for those who use Terraform or other IaC tooling?

You wouldn't use aspire for deployment. You would continue to use terraform (until somebody writes a manifest -> terraform translation layer).

[normj]

What about for those who use Terraform or other IaC tooling?

I would leave Terraform up to the fine folks at Hashicorp or the community. I know Terraform is popular but I have very little experience with it as I have a bias for AWS tools and services 😄. That being said if there are any extensions or mechanism we need to make to help a Terraform component I happy to help.

[davidfowl]

OK I think I like this low-level model. High level comments:

• IAWSSDKConfigResource, I don't think this should be a resource, it feels like it should just be configuration.
• WithAWSSDKReference and WithAWSCloudFormationReference I would rename to WithReference and rely on overload resolution (it'll be consistent with the other APIs).
• Who defines how outputs get mapped to environment?
• I think we would need to flesh out what this resource looks like in the manifest as well. Maybe we'll end up with a single cloudformation.v0 resource type.
• There could be a single CloudFormationLifecycleHook that handles all cloud formation resources (instead of one per resource)

[davidfowl]

Suggested change

	var chatTopicArn = builder.Configuration.GetSection("AWS:Resources")["ChatTopicArn"];
	var chatTopicArn = builder.Configuration["AWS:Resources:ChatTopicArn"];

This is one of the things we're going to have to wrangle with aspire. Passing connection information in a consistent way and consuming it in a consistent way is part of the opinionated stack and what makes it easy to use.

The low-level infrastructure is fine, but I hope we can build something a bit more like the other components use a single config section with a fixed schema per component.

[normj]

Updated the code.

The configuration section will be hard to have a fixed schema since the output of a CloudFormation stack is dynamic to whatever the user defines in the template.

[davidfowl]

That breaks the ease of use in the application code. We want free form config to work as well. but part of the value of aspire beyond the orchestration is having a component on the other side that picks up this configuration and does something with it.

Consider this example that uses mysql in the app host:

var builder = DistributedApplication.CreateBuilder(args);

var db = builder.AddMySql("server").AddDatabase("db");

builder.AddProject<Projects.WebApplication1>("api")
    .WithReference(db);

builder.Build().Run();

Then consumes it in the app using the same names:

using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();
builder.AddMySqlDbContext<AppDb>("db"); // pick up the db configuration

var app = builder.Build();

app.MapGet("/", async (AppDb db) =>
{
    await db.Database.EnsureCreatedAsync();

    db.Entries.Add(new Entry { Name = Guid.NewGuid().ToString() });
    await db.SaveChangesAsync();

    return await db.Entries.ToListAsync();
});

app.Run();

[normj]

I removed the extra CloudFormationResourceReference resource and now using the CloudFormationReferenceAnnotation which is attached to the CloudFormation resource. My initial hang up with this approach is I wanted to add the reference on the Project resource but I couldn't see how to add custom annotations to the manifest for a project so I fell back to creating the reference resource. Your comment turn the light bulb on my head that I could add the list of references on the CloudFormation resource which I control the manifest entry for.

Here is an example of what the manifest looks like now. The references section is a list of objects instead of resource names to give myself room for future additions.

{
  "resources": {
    "AspireSampleDevResources": {
      "type": "aws.cloudformation.template.v0",
      "stack-name": "AspireSampleDevResources",
      "template-path": "app-resources.template",
      "references": [
        {
          "TargetResource": "Frontend"
        }
      ]
    },
    "Frontend": {
      "type": "project.v0",
      "path": "../Frontend/Frontend.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ChatTopicArnEnv": "{AspireSampleDevResources.aws.cloudformation.output.ChatTopicArn}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    }
  }
}

[davidfowl]

Here is an example of what the manifest looks like now. The references section is a list of objects instead of resource names to give myself room for future additions.

Yes! Also I can tell you that azd parses the env section for expressions and uses it to determine dependencies. It's more implicit, but it works well. In the above manifest, the "FrontEnd" resource reference's env section references "AspireSampleDevResources"'s outputs so there's a dependency between FrontEnd and AspireSampleDevResources.

        "ChatTopicArnEnv": "{AspireSampleDevResources.aws.cloudformation.output.ChatTopicArn}"

[normj]

This PR is ready for review. The Directory.Packages.props has been updated with the latest AWS dependencies.

[eerhardt]

These's this build error as well:

This should be fixed now. It builds locally for me.

[davidfowl]

Thank you @normj !