Could we get an example of a Redis Elasticache addon?

[DennisSSDev]

Hi

I wanted to make a caching layer for my backend (private) service using elasticache + redis.

Could we get an example of a simple addon yaml file that creates a 50mb redis elasticache instance that respects the default copilot backend service private network setup?

All other settings such as replication can be assumed to be whatever you want, I just want to see how the most barebones elastic cache redis addon file looks like 😄

[efekarakus]

Hi @DennisSSDev !

Sorry this is a bit late but here is a working template for using a private elasticache cluster!
https://gist.github.com/efekarakus/fe694581c88e03eedb38f17b8ddc68a7

Addon template for Redis

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.

Resources:
  # Subnet group to control where the Redis gets placed
  RedisSubnetGroup:
    Type: AWS::ElastiCache::SubnetGroup
    Properties:
      Description: Group of subnets to place Redis into
      SubnetIds: !Split [ ',', { 'Fn::ImportValue': !Sub '${App}-${Env}-PrivateSubnets' } ]
  
  # Security group to add the Redis cluster to the VPC,
  # and to allow the Fargate containers to talk to Redis on port 6379
  RedisSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "Redis Security Group"
      VpcId: { 'Fn::ImportValue': !Sub '${App}-${Env}-VpcId' }
  
  # Enable ingress from other ECS services created within the environment.
  RedisIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Ingress from Fargate containers
      GroupId: !Ref 'RedisSecurityGroup'
      IpProtocol: tcp
      FromPort: 6379
      ToPort: 6379
      SourceSecurityGroupId: { 'Fn::ImportValue': !Sub '${App}-${Env}-EnvironmentSecurityGroup' }

  # The cluster itself.
  Redis:
    Type: AWS::ElastiCache::CacheCluster
    Properties:
      Engine: redis
      CacheNodeType: cache.m4.large
      NumCacheNodes: 1
      CacheSubnetGroupName: !Ref 'RedisSubnetGroup'
      VpcSecurityGroupIds:
        - !GetAtt 'RedisSecurityGroup.GroupId'

  # Redis endpoint stored in SSM so that other services can retrieve the endpoint.
  RedisEndpointAddressParam:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub '/${App}/${Env}/${Name}/redis'   # Other services can retrieve the endpoint from this path.
      Type: String
      Value: !GetAtt 'Redis.RedisEndpoint.Address'

Outputs:
  RedisEndpoint:
    Description: The endpoint of the redis cluster
    Value: !GetAtt 'Redis.RedisEndpoint.Address'

This will create an elasticache cluster that's available to all your Copilot services in your VPC.

If you want other services to access this cluster you can inject the endpoint in the manifest like this:

secrets:
  REDIS_ENDPOINT: /{app}/{env}/{main service name}/redis

Hope this helps!

[sugarjig]

Ah. I tried the exact template in the example, and that worked. I had made some modifications that were apparently not valid. Thanks for your help!

[TillaTheHun0]

Thanks for this example! A couple questions.

Adding RedisSecurityGroup.GroupId to Redis.Properties.VpcSecurityGroupIds adds Redis to the VPC, but doesn't set any Ingress/Egress for Redis, as that is what RedisIngress is doing, correct?

Per the Copilot docs:

If you need to add a security group to your ECS service, you can define a Security Group in your template, and then add it as an Output. The security group will be automatically attached to your ECS service.

Based on that, if I wanted only the ECS service that this Redis "addon" is nested under, to be able to consume the Redis resource, would I create a security group resource, add it as the RedisIngress.Properties.SourceSecurityGroupId, and then add that same security group as an Output?

So perhaps somethings like:

...
Resources:
  # Add this
  RedisSourceSecurityGroup:
    Metadata:
      'aws:copilot:description': 'An EC2 Security Group to add to our ECS Service, in order to consume Redis'
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "Redis Consumer Security Group"
      VpcId: { 'Fn::ImportValue': !Sub '${App}-${Env}-VpcId' }

  RedisSecurityGroup:
    Metadata:
      'aws:copilot:description': 'An EC2 Security Group to add the Redis cluster to the environment VPC and enable ingress from our ECS Service'
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "Redis Security Group"
      VpcId: { 'Fn::ImportValue': !Sub '${App}-${Env}-VpcId' }
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 6379
          ToPort: 6379
          SourceSecurityGroupId: !GetAtt 'RedisSourceSecurityGroup.GroupId' # Ref new security group here

...
Outputs:
  RedisEndpoint:
    Description: The endpoint of the redis cluster
    Value: !GetAtt 'Redis.RedisEndpoint.Address'
  # Add as an output and Copilot adds to our ECS Service
  RedisSourceSecurityGroup:
    Description: "The EC2 Security Group to add to our ECS Service, in order to consume Redis."
    Value: !Ref RedisSourceSecurityGroup

Edit: added example

[efekarakus]

Hi @TillaTheHun0 !

Adding RedisSecurityGroup.GroupId to Redis.Properties.VpcSecurityGroupIds adds Redis to the VPC, but doesn't set any Ingress/Egress for Redis, as that is what RedisIngress is doing, correct?

Correct!

Based on that, if I wanted only the ECS service that this Redis "addon" is nested under, to be able to consume the Redis resource, would I create a security group resource,

Also correct 😄

What you have looks good. The SecurityGroup output will indeed add the security group to only your service and not allow it to any other services.
For the ingress, you'll want to change this:

SourceSecurityGroupId: { 'Fn::ImportValue': !Sub '${App}-${Env}-EnvironmentSecurityGroup' }

So that it doesn't allow traffic from all the containers in your environment, but instead I think to only RedisSecurityGroup

[TillaTheHun0]

@efekarakus great, thanks! I created a new security group RedisSourceSecurityGroup, and used that as the SourceSecurityGroupId and Output, so that my ECS service would not have the ingress rules for RedisSecurityGroup applied to it. I hope I have that right (see example above)

Thanks again!

[efekarakus]

That looks good!

[deodad]

I tried to run this example and am running into the following error when the service stack tries to update (the add on stack is successfully setting up):

Resource handler returned message: "Invalid request provided: UpdateService error: portName(target) does not refer to any named PortMapping in the container definitions. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 38d67ab3-8964-4923-8197-cc3e2b470c49; Proxy: null)" (RequestToken: 8226609c-a3f7-abac-bf3b-1193ea3e990a, HandlerErrorCode: InvalidRequest)

Any ideas?