Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

App model rework around resource types and connection string emission for containers #1165

Merged
merged 14 commits into from
Dec 7, 2023

Conversation

mitchdenny
Copy link
Member

@mitchdenny mitchdenny commented Dec 1, 2023

Fixes:
#1164
#1088

This PR is a fairly major change to all the resource types built into Aspire.Hosting to support a model where calls like AddRedisContainer actually outputs a container resource to the manifest (container.v0) with enough configuration such that it can run within a given orchestrator.

It introduces three new key concepts:

  1. Container resources can now explicitly declare a connectionString field in the manifest. When a resource connection string is referenced this field can be used.
  2. Because the content of connectionString cannot be determined at manifest publishing time it now contains placeholder values that the deployment tool is expected to process and replace as appropriate.
  3. There is a new per resource field inputs (optional) which tells the deployment tool when it should prompt the user (or generate) inputs such as a password.

The following JSON manifest is produces from tests/testproject/TestProject.AppHost and contains the built-in resource types that employ this model.

Note that as part of this change, we have removed AddXYZConnection(...) methods and associated resource types as we found they added minimal value beyond just adding an environment variable. Also note that we've added a non-container version of most of the resource types (e.g. AddRedis(...) vs. AddRedisContainer(...)). Locally these methods do much the same thing, but when the manifest is produced the first option does not produce a container.v0 resource, it produces redis.v0 like it previously did.

Ideally folks will use these more abstract builder methods (e.g. AddRedis(...)) instead of the container orientated counterpart. Deployment tools that target cloud environments should preference the use of managed services with better operational support, particularly in production deployment scenarios.

Deployment tools may wish to warn users that they are using container builder methods vs. abstract resource type builder methods when deploying to production, although the containerized versions should still continue to work.

{
  "resources": {
    "servicea": {
      "type": "project.v0",
      "path": "../TestProject.ServiceA/TestProject.ServiceA.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "serviceb": {
      "type": "project.v0",
      "path": "../TestProject.ServiceB/TestProject.ServiceB.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "servicec": {
      "type": "project.v0",
      "path": "../TestProject.ServiceC/TestProject.ServiceC.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "workera": {
      "type": "project.v0",
      "path": "../TestProject.WorkerA/TestProject.WorkerA.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true"
      }
    },
    "nodeapp": {
      "type": "executable.v0",
      "workingDirectory": "../nodeapp",
      "command": "node",
      "args": [
        "C:\\Code\\dotnet\\aspire\\tests\\testproject\\TestProject.AppHost\\..\\nodeapp\\app.js"
      ],
      "env": {
        "NODE_ENV": "production",
        "PORT": "{nodeapp.bindings.http.port}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "npmapp": {
      "type": "executable.v0",
      "workingDirectory": "../nodeapp",
      "command": "npm",
      "args": [
        "run",
        "start"
      ],
      "env": {
        "NODE_ENV": "production",
        "PORT": "{npmapp.bindings.http.port}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "sqlservercontainer": {
      "type": "container.v0",
      "image": "mcr.microsoft.com/mssql/server:2022-latest",
      "env": {
        "ACCEPT_EULA": "Y",
        "MSSQL_SA_PASSWORD": "{sqlservercontainer.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 1433
        }
      },
      "connectionString": "Server={sqlservercontainer.bindings.tcp.host},{sqlservercontainer.bindings.tcp.port};User ID=sa;Password={sqlservercontainer.inputs.password};TrustServerCertificate=true;",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "mysqlcontainer": {
      "type": "container.v0",
      "image": "mysql:latest",
      "env": {
        "MYSQL_ROOT_PASSWORD": "{mysqlcontainer.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 3306
        }
      },
      "connectionString": "Server={mysqlcontainer.bindings.tcp.host};Port={mysqlcontainer.bindings.tcp.port};User ID=root;Password={mysqlcontainer.inputs.password}",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "rediscontainer": {
      "type": "container.v0",
      "image": "redis:latest",
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 6379
        }
      },
      "connectionString": "{rediscontainer.bindings.tcp.host}:{rediscontainer.bindings.tcp.port}"
    },
    "postgrescontainer": {
      "type": "container.v0",
      "image": "postgres:latest",
      "env": {
        "POSTGRES_HOST_AUTH_METHOD": "scram-sha-256",
        "POSTGRES_INITDB_ARGS": "--auth-host=scram-sha-256 --auth-local=scram-sha-256",
        "POSTGRES_PASSWORD": "{postgrescontainer.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 5432
        }
      },
      "connectionString": "Host={postgrescontainer.bindings.tcp.host};Port={postgrescontainer.bindings.tcp.port};Username=postgres;Password={postgrescontainer.inputs.password};",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "rabbitmqcontainer": {
      "type": "container.v0",
      "image": "rabbitmq:3-management",
      "env": {
        "RABBITMQ_DEFAULT_USER": "guest",
        "RABBITMQ_DEFAULT_PASS": "{rabbitmqcontainer.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 5672
        },
        "management": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http",
          "containerPort": 15672
        }
      },
      "connectionString": "amqp://guest:{rabbitmqcontainer.inputs.password}@{rabbitmqcontainer.bindings.management.host}:{rabbitmqcontainer.bindings.management.port}",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "mongodbcontainer": {
      "type": "container.v0",
      "image": "mongo:latest",
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 27017
        }
      },
      "connectionString": "{mongodbcontainer.bindings.tcp.host}:{mongodbcontainer.bindings.tcp.port}"
    },
    "sqlserverabstract": {
      "type": "container.v0",
      "image": "mcr.microsoft.com/mssql/server:2022-latest",
      "env": {
        "ACCEPT_EULA": "Y",
        "MSSQL_SA_PASSWORD": "{sqlserverabstract.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 1433
        }
      },
      "connectionString": "Server={sqlserverabstract.bindings.tcp.host},{sqlserverabstract.bindings.tcp.port};User ID=sa;Password={sqlserverabstract.inputs.password};TrustServerCertificate=true;",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "mysqlabstract": {
      "type": "container.v0",
      "image": "mysql:latest",
      "env": {
        "MYSQL_ROOT_PASSWORD": "{mysqlabstract.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 3306
        }
      },
      "connectionString": "Server={mysqlabstract.bindings.tcp.host};Port={mysqlabstract.bindings.tcp.port};User ID=root;Password={mysqlabstract.inputs.password}",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "redisabstract": {
      "type": "container.v0",
      "image": "redis:latest",
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 6379
        }
      },
      "connectionString": "{redisabstract.bindings.tcp.host}:{redisabstract.bindings.tcp.port}"
    },
    "postgresabstract": {
      "type": "container.v0",
      "image": "postgres:latest",
      "env": {
        "POSTGRES_HOST_AUTH_METHOD": "scram-sha-256",
        "POSTGRES_INITDB_ARGS": "--auth-host=scram-sha-256 --auth-local=scram-sha-256",
        "POSTGRES_PASSWORD": "{postgresabstract.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 5432
        }
      },
      "connectionString": "Host={postgresabstract.bindings.tcp.host};Port={postgresabstract.bindings.tcp.port};Username=postgres;Password={postgresabstract.inputs.password};",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "rabbitmqabstract": {
      "type": "container.v0",
      "image": "rabbitmq:3-management",
      "env": {
        "RABBITMQ_DEFAULT_USER": "guest",
        "RABBITMQ_DEFAULT_PASS": "{rabbitmqabstract.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 5672
        },
        "management": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http",
          "containerPort": 15672
        }
      },
      "connectionString": "amqp://guest:{rabbitmqabstract.inputs.password}@{rabbitmqabstract.bindings.management.host}:{rabbitmqabstract.bindings.management.port}",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "mongodbabstract": {
      "type": "mongodb.server.v0"
    },
    "integrationservicea": {
      "type": "project.v0",
      "path": "../TestProject.IntegrationServiceA/TestProject.IntegrationServiceA.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__sqlservercontainer": "{sqlservercontainer.connectionString}",
        "ConnectionStrings__mysqlcontainer": "{mysqlcontainer.connectionString}",
        "ConnectionStrings__rediscontainer": "{rediscontainer.connectionString}",
        "ConnectionStrings__postgrescontainer": "{postgrescontainer.connectionString}",
        "ConnectionStrings__rabbitmqcontainer": "{rabbitmqcontainer.connectionString}",
        "ConnectionStrings__mongodbcontainer": "{mongodbcontainer.connectionString}",
        "ConnectionStrings__sqlserverabstract": "{sqlserverabstract.connectionString}",
        "ConnectionStrings__mysqlabstract": "{mysqlabstract.connectionString}",
        "ConnectionStrings__redisabstract": "{redisabstract.connectionString}",
        "ConnectionStrings__postgresabstract": "{postgresabstract.connectionString}",
        "ConnectionStrings__rabbitmqabstract": "{rabbitmqabstract.connectionString}",
        "ConnectionStrings__mongodbabstract": "{mongodbabstract.connectionString}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    }
  }
}

@dotnet-issue-labeler dotnet-issue-labeler bot added the area-codeflow for labeling automated codeflow. intentionally a different color! label Dec 1, 2023
@mitchdenny
Copy link
Member Author

ping @ellismg @davidfowl @prom3theu5

@mitchdenny mitchdenny added area-app-model Issues pertaining to the APIs in Aspire.Hosting, e.g. DistributedApplication and removed area-codeflow for labeling automated codeflow. intentionally a different color! labels Dec 1, 2023
@mitchdenny mitchdenny self-assigned this Dec 1, 2023
@mitchdenny mitchdenny added this to the preview 2 (Dec) milestone Dec 1, 2023
@mitchdenny
Copy link
Member Author

Made some changes for Postgres. Specifically, we no longer use the trust auth method either locally for for inner loop. Instead with use the scram-sha-256 password exchange method. It stops the password being transmitted via clear text (using a challenge mechanism). The connection still isn't TLS though - but its better than auth mode being trust if someone is deploying onto an internal container network.

@mitchdenny
Copy link
Member Author

@davidfowl @ellismg @prom3theu5 ... big push! Let me hear your thoughts and feedback.

@davidfowl
Copy link
Member

Why does the nodeapp have the full physical path to app.js, that seems broken. Are we resolving the full path and putting it in args? I'll look into that.

One thing we should consider is not having containers require a custom manifest but instead using annotations. That would let the manifest writing happen without so much custom logic in each container (I worry about the amount of copy pasta required for new containers)

@prom3theu5
Copy link

It does look better now.

Quite a lot of rework in aspirate for third party component support for things like Postgres and Redis now though, but hopefully the amount of data about what comprises a container in the manifest is enough to construct custom deployments for them etc

The different there is when using the component type it was each to have a custom processor for that component that pulled separate templates for the components.

So for redis it deployed a stateful set of a standard redis instance based on a separate template. for Postgres it wasn't a stateful set but a deployment with custom health checks etc, and the "processors" were handled by the component type.

I'll make it work though, I agree this is a change for the better.

@davidfowl
Copy link
Member

Quite a lot of rework in aspirate for third party component support for things like Postgres and Redis now though, but hopefully the amount of data about what comprises a container in the manifest is enough to construct custom deployments for them etc

Why? Those resources are still abstract resources that the deployment tool needs to handle. This PR just makes container resources more explanatory.

@davidfowl
Copy link
Member

@mitchdenny do you plan to update the templates to use AddRedis instead of AddRedisContainer?

@prom3theu5
Copy link

prom3theu5 commented Dec 6, 2023

Just because of how I implemented components is all

Each component had its own processor like https://github.com/prom3theu5/aspirational-manifests/blob/main/src/Aspirate.Processors/Redis/RedisProcessor.cs

I agree it simplifies a lot of that just needs a more “intelligent” processor for the container type.

The way I implemented component types was with their own processors and own set of templates so the processor for redis It explicitly chose a redis template with lower resource limits set etc. now each type will just be container.v0

https://github.com/prom3theu5/aspirational-manifests/blob/main/src/Aspirate.Cli/Templates/redis.hbs

I can use the type to control the selection of which template. Now on a container if you change the custom image for redis to joeblogs:about-stuff there is no way to know that it's redis.

Components like redis don't technically need a service as well as a deployment defining.

Don't worry. I'll just produce a container processor and rework all the custom templates into one unified templated template. Worst case everything will just end up with a matching service.

Besides this container type is in addition to the other resource types isn't it?
So it's technically not a rework just another resource type that produces a fully custom deployment.

@mitchdenny
Copy link
Member Author

@davidfowl up for discussion. I think it would prefer to use the non container variants.

@prom3theu5 keep your existing logic for redis etc, those resource types still exist. The sample should be updated to use them.

@mitchdenny
Copy link
Member Author

@davidfowl template updated.

@prom3theu5 I've updated the sample AppHost to go back to using the abstract resource types (AddRedis instead of AddRedisContainer). This means the sample manifest outputs the following which should be exactly the same as what you are used to dealing with:

{
  "resources": {
    "postgres": {
      "type": "postgres.server.v0"
    },
    "catalogdb": {
      "type": "postgres.database.v0",
      "parent": "postgres"
    },
    "basketcache": {
      "type": "redis.v0"
    },
    ...

The changes made in this PR mean that when you do something like AddRedisContainer you'll get this:

{
  "resources": {
    "postgres": {
      "type": "postgres.server.v0"
    },
    "catalogdb": {
      "type": "postgres.database.v0",
      "parent": "postgres"
    },
    "basketcache": {
      "type": "container.v0",
      "image": "redis:latest",
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 6379
        }
      },
      "connectionString": "{basketcache.bindings.tcp.host}:{basketcache.bindings.tcp.port}"
    },
    ...

Note that the basketcache resource which was redis.v0 has been replaced with a regular container.v0 resource. Deployment tools should process these like a regular container (which was already an existing resource). The key difference is the addition of the connectionString field to a container (optiona) and the inputs field (also optional).

@mitchdenny
Copy link
Member Author

mitchdenny commented Dec 6, 2023

One thing we should consider is not having containers require a custom manifest but instead using annotations. That would let the manifest writing happen without so much custom logic in each container (I worry about the amount of copy pasta required for new containers)

We are already pretty late, so if you are happy to do so can we defer that to preview 3. I think we would probably need an ManifestInputAnnotation and a ManifestConnectionStringAnnotation.

@davidfowl
Copy link
Member

We are already pretty late, so if you are happy to do so can we defer that to preview 3. I think we would probably need an ManifestInputAnnotation and a ManifestConnectionStringAnnotation.

Lets file a follow up issue. This needs to be backported to preview2.

Also, this is a breaking change since everyone is using AddRedisContainer today 😄

@mitchdenny mitchdenny merged commit 1ae3c40 into main Dec 7, 2023
8 checks passed
@mitchdenny mitchdenny deleted the abstract-resources branch December 7, 2023 05:59
mitchdenny added a commit that referenced this pull request Dec 7, 2023
… for containers (#1165)

* Abstract resources and templated connection strings.

* Add abstract redis for testing.

* Moved MySql to input request model.

* Postgres implementation.

* Update PostgresBuilderExtensions.cs

Co-authored-by: Eric Erhardt <[email protected]>

* Migrate RabbitMQ to new model.

* Migrate SQL to new model.

* Test case updates.

* Update mongo resource to the new model.

* Update manfiests.

* Update template.

---------

Co-authored-by: Eric Erhardt <[email protected]>
@mitchdenny
Copy link
Member Author

@davidfowl backport PR: #1270

davidfowl pushed a commit that referenced this pull request Dec 8, 2023
… for containers (#1165) (#1270)

* Abstract resources and templated connection strings.

* Add abstract redis for testing.

* Moved MySql to input request model.

* Postgres implementation.

* Update PostgresBuilderExtensions.cs



* Migrate RabbitMQ to new model.

* Migrate SQL to new model.

* Test case updates.

* Update mongo resource to the new model.

* Update manfiests.

* Update template.

---------

Co-authored-by: Eric Erhardt <[email protected]>
{
var sqlServerConnection = new SqlServerConnectionResource(name, connectionString);
var password = Guid.NewGuid().ToString("N") + Guid.NewGuid().ToString("N").ToUpper();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason to repeat the guid?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason to repeat the GUID is that the default password Policy for the SQL container needs upper case and lower case characters with special cases. This is just a cheats way of using GUIDs to do that.

@sebastienros
Copy link
Member

@mitchdenny The sql client doesn't work anymore after this PR, the password is invalid for sa when resolving SqlConnection. Or is there something else to do than AddDatabase() in the host and AddSqlServerClient in the app?

@mitchdenny
Copy link
Member Author

What does your app host look like so I can repro.

@mitchdenny
Copy link
Member Author

I am not sure there is a problem here, I just tested it with tests/testprogram/TestProgram.AppHost. Here was my methodology:

  1. Start the app host.
  2. Go to integration service /health endpoint and verify that everything is healthy.
  3. Go to Docker Desktop and stop the SQL server instances (there are two of them)
  4. Hit the health endpoint again ... should now be unhealthy.
  5. Start both containers again ... should eventually go healthy again.

Comment on lines +41 to +46
var sqlserverAbstract = AppBuilder.AddSqlServerContainer("sqlserverabstract");
var mysqlAbstract = AppBuilder.AddMySqlContainer("mysqlabstract");
var redisAbstract = AppBuilder.AddRedisContainer("redisabstract");
var postgresAbstract = AppBuilder.AddPostgresContainer("postgresabstract");
var rabbitmqAbstract = AppBuilder.AddRabbitMQContainer("rabbitmqabstract");
var mongodbAbstract = AppBuilder.AddMongoDB("mongodbabstract");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mitchdenny - were these "Abstract" resources all not supposed to have Container on them? As this stands, this code spins up 2 SQL Server containers in the exact same way. Same for MySql, Redis, Postgres, RabbitMQ.

@github-actions github-actions bot locked and limited conversation to collaborators Apr 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-app-model Issues pertaining to the APIs in Aspire.Hosting, e.g. DistributedApplication
Projects
None yet
6 participants