[Fargate] [Volumes]: Allow at least EFS mounts to Fargate Containers

[archisgore]

Tell us about your request
Allow mounting of at least EFS volumes (if nothing more generic or extensible) onto Fargate tasks.

Which service(s) is this request for?
Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We're in en-masse plans to migrate to Fargate. We're 100% in agreement with having stateless containers talking to stable external storage over the network (S3, DynamoDB). We'll do it sooner or later. You won't lose business without this.

This is an empathetic ask - if we could mount at LEAST EFS volumes to support those external workloads (stuff we don't build, but rather download), then it allows a large life-and-shift to Fargate, getting rid of Docker for AWS and ECS and gives us one consistent team-wide technology to consume, while we the factor out those dependencies cleanly.

Are you currently working around this issue?
We use Docker Swarm using the old DFA CloudFormation stack. Looked into ECS before volume plugins were supported and just the 2-3 level steps was awful (create volume, mount on host, remember where it is on host, launch task, mount directory to volume, mount volume to container.)

[FernandoMiguel]

ECS now allows for volumes to be mounted at task level, not host only..
Check it out.

[casperc]

@FernandoMiguel : Does that mean that EFS is now supported by Fargate?

[Akramio]

Thanks everyone for this request. It would really be awesome if you could give us a little more detail about your need for this feature: For example, which workloads / applications that require EFS would you want to deploy on ECS? Would also love to hear about any potential use-cases or interests in using the newly released FSx file system.

[jonathonsim]

We'd very much like to see support for EFS in fargate. I imagine there's a multitude of applications - the one we have is that we (Idealstack) are doing website hosting in ECS, and want to support common PHP-based web apps such as wordpress, drupal, peoples custom PHP code etc. These typically require shared storage if you want to cluster them and autoscale, and don't support S3 in general. So for instance the AWS reference architecture for wordpress, magento or drupal all use EFS

But I would imagine in any situation where users want persistant storage wth unix filesystem semantics EFS is going to be helpful, particularly where you are moving existing apps into fargate. There's a lot of frustration on the internet over the lack of EFS support in fargate dating to when it was first released. FSx filesystems aren't something we use, but would probably also have similar applications, as would EBS support. Something similar to the new volume driver support for (non-fargate) ECS would be great, even if only certain AWS-managed drivers that supported a few common targets such as EFS, EBS and/or FSx were supported

With EFS in fargate we could support more effective autoscaling of these kinds of apps compared to EC2-based architectures (since a container can boot in seconds versus minutes to create an instance and add a container instance in ECS). This would be a killer feature for our product. Lack of EFS support stops us from supporting Fargate at the moment though.

[lifeofguenter]

adding to @jonathonsim -there are quite some open-source services that are not built cloud-native:

• jenkins (the primary/master, not nodes)
• grafana (or many other graphing tools)
• logstash/beats

Many of those tools just need a persistent storage for minimal writes.

[archisgore]

As original author, I'll give you some my use-cases:

1. Private Docker registries (unless ECR allows us to host public/private repos for external distribution) like Harbor or Nexus. They can store blobs to S3, but still need a filesystem for state/config.
2. Other legacy software that reads/writes config/state to/from disk - Java Webservers, etc.
3. SQL databases. I know, I know, Aurora and RDS, but just trust me on this. We are a cybersecurity company, so we need to occasionally host containerized databases to host wordpress against. EFS would allow the database to be persistent, but Fargate would allow us to test various SQL injection scenarios against it and mitigations.
4. Wordpress plugins generate new PHP code on the fly. This can't go to S3. Can't go to DynamoDB. Needs to be persisted under a filesystem.

The counter-pressure on why NOT to use ECS: That escape-hatch becomes an opportunity for hard-work-creep. Opens up new AMIs, custom Linuxes, host drivers, firewalls, authentication, and more. It's too much opportunity opened up only to give someone the ability to mount stage storage so their wordpress plugin can generate code on the fly.

[ahammond]

Our use case would be mounting a volume read-only which contains static data (in our case the reference genome) instead of having to put this data in the image or download it from S3 every time we start a container.

[peterfranzen]

This would be extremely helpful. We have a task that we'd like to run in Fargate that currently involves pulling around 30GB of data in from S3 each time it runs; we can do this in EC2 or on ECS containers, but it would save us a ton of headaches if we could load it directly into a Fargate volume.

[JoseRolles]

Yes, this would be awesome! Even this AWS whitepaper on WordPress "best practices" in the "stateless web tier" references using EFS to store plugin files https://d1.awsstatic.com/whitepapers/wordpress-best-practices-on-aws.pdf

Exactly what we are trying to achieve with Fargate. This would allow [plugin/cms] updates to happen on the fly [by WP admins].

[soukicz]

Exactly what we are trying to achieve with Fargate. This would allow updates to happen on the fly.

That is actually something that you don't want because than you don't hava atomic deployment.

But EFS in fargate does have use cases. For example shared file cache for compiled templates.

[JoseRolles]

What we need is for WordPress admins to be able to add plugins on their own. Even the whitepaper suggests this in a "stateless web tier".

[adriantunez]

+1 here, we've been waiting for this feature since Fargate was launched. We have lots of applications with a high ratio of connections waiting to be migrated into Fargate to make use of autoscaling feature but we need to be able to mount the same EFS to these services in order to share some information between containers

[adriantunez]

ECS now allows for volumes to be mounted at task level, not host only..
Check it out.

That feature isn't supported currently by Fargate.

In addition @abby-fuller IMHO it should be labelled as Fargate.

[dambrogia]

Having EFS + fargate would allow me to skim off 5-6 minutes of my deployments. It would also provide extra flexibility/agility for adjusting configs during triage time.

I could deploy to the EFS mount with the ability of instant upgrades/rollbacks through something like deployer rather than needing to build a new docker image + deploying that to an ASG/Cluster with hardcoded configs built into them and wait for the replace action to occur in cloud formation.

[cgswong]

Our use cases involve running stateful workloads, such as cluster state (software requires file systems), custom databases, and CloudFoundry migration -- for workloads requiring file systems.

Would love this feature, though we are looking at a more mature EKS as well.

[skwokie]

Hi, I'd like to ask if there is an ETA for this. Fargate is one of the platforms that we are looking into for migrating our service and NFS/EFS support is a very important feature that our service uses; and knowing the ETA will be very helpful for planning our schedule. Thanks.

[ghost]

It would be really helpful to know more about EFS/NFS volume mounting on containers running on Fargate, and if this is going to be implemented in the near future. I am currently looking for a solution to connect jupyterhub to fargate. For data scientists to save their notebooks in their jupyter instance (running in the container), we need to mount a volume. So they can continue their work the next time they log in. Other options would need us to keep a large EC2 instance running all the time. This would cost a lot for every new user.

[larryboymi]

It would be really helpful to know more about EFS/NFS volume mounting on containers running on Fargate, and if this is going to be implemented in the near future. I am currently looking for a solution to connect jupyterhub to fargate. For data scientists to save their notebooks in their jupyter instance (running in the container), we need to mount a volume. So they can continue their work the next time they log in. Other options would need us to keep a large EC2 instance running all the time. This would cost a lot for every new user.

I'm doing something very similar with RStudio...

[ghost]

AWS says they are working on EFS support for ECS with Fargate, if I may believe this post: https://forums.aws.amazon.com/thread.jspa?messageID=816397&tstart=0

[mi-hol]

this feature would allow for rehosting of existing applications with minimal effort. Desperately required!

[pauldraper]

I would like to operate a FTPS/SFTP server with scalable/durable storage.

[adriantunez]

Do we have any ETA for that??

[teamfighter]

We really need this feature. ECS at EC2 is a headache - autoscaling groups, cloudinit, mount directory to host, mount to task... Awful.

[FernandoMiguel]

ECS at EC2 is a headache - autoscaling groups, cloudinit, mount directory to host, mount to task... Awful.

@teamfighter awful? was 4 lines of code for efs, and another 4 for the task definition

[teamfighter]

@FernandoMiguel I didnt told that it's impossible. I told that it is uncomfortable solution.

[ghost]

Has anyone tried to mount an s3 bucket inside a container running on fargate with s3fs? This may be a (temporary) solution to persist files to s3. I am currently using s3fs to mount/share files between ec2 instances, and it works like a charm!

[FernandoMiguel]

Has anyone tried to mount an s3 bucket inside a container running on fargate with s3fs? This may be a (temporary) solution to persist files to s3. I am currently using s3fs to mount/share files between ec2 instances, and it works like a charm!

pretty PLEASE don't use s3fs.... S3 is object storage... trying to treat it as persistent storage is a terrible idea @juultje123

[mustanggb]

Excellent news, thanks for the update.

[surajtikoo]

Amazing work.After a long time. Happy to hear

[ChristianHartTE]

Thank you @coultn! Been waiting for this one for a while.

[soukicz]

Great news!
Looks like you have to manually change platform version on ecs service to 1.4.0 - it is using 1.3.0 if I keep "LATEST". But I might have been just too fast :)

[mikesir87]

@coultn After reading the blog post, it suggests that this support is for ECS-only workloads, to cover both EC2 and Fargate launch types. No mention is made of Fargate workloads in EKS. So, I'm assuming that's out of scope for this release? If so, that's fine. Just trying to clarify because I also see the EKS label on this issue and the issue was closed.

[mildebrandt]

Great! Is there a corresponding item to follow for CloudFormation support?

[mreferre]

@coultn After reading the blog post, it suggests that this support is for ECS-only workloads, to cover both EC2 and Fargate launch types. No mention is made of Fargate workloads in EKS. So, I'm assuming that's out of scope for this release? If so, that's fine. Just trying to clarify because I also see the EKS label on this issue and the issue was closed.

You are correct in your assumption @mikesir87. We are working to enable this scenario for EKS. Stay tuned.

[srrengar]

Opened an issue for CloudFormation support
#825

[mikesir87]

We are working to enable this scenario for EKS. Stay tuned.

Is there an issue that we can follow for that support @mreferre?

[hlascelles]

@mreferre Thank you for delivering this for ECS.

I'm glad to hear EKS support is on the way too; I have created a new issue to track that specifically: #826

[pixelicous]

Amazing that EFS got to be supported, what about EBS? 😢 @coultn

[synth]

Is this possible to configure with CloudFormation?

[mreferre]

Is this possible to configure with CloudFormation?

@synth not yet but we are working on getting that support shipped asap. We do know it is in high demand. Stay tuned.

[EthanNguyenDev]

good news! this's what we're looking for to migrate some of our existing stateful workload to Fargate

[sourabh-idoorkar]

Is this possible to configure with CloudFormation?

@synth not yet but we are working on getting that support shipped asap. We do know it is in high demand. Stay tuned.

Have been trying to find a way to get this working and only now found this thread. Now it makes sense why it has been failing for me from CloudFormation. Eagerly awaiting this feature in CloudFormation

[uherberg]

@mreferre Is there an ETA for Cloudformation support of EFS for Fargate? We are eagerly waiting for that feature

[mreferre]

@mreferre Is there an ETA for Cloudformation support of EFS for Fargate? We are eagerly waiting for that feature

@uherberg we are actively working on it. I don't have more details to share at this time. Stay tuned. We will update this post when CloudFormation support is introduced. Thanks for your patient.

[hlarsen]

are we able to use EFS mounts from other accounts?

we have a pair of VPCs peered and mounting the EFS share cross-account works on ec2 (after fixing DNS by specifying the IP of the mount in /etc/hosts), but for Fargate we're only passing the mount name.

[mreferre]

@hlarsen this won't work because of DNS resolution. It would work with a shared VPC among the two accounts though but not with two separate VPCs. Can you open a new GH issue with this specific request so that we can track it? Thanks.

[babaMar]

@mreferre Is there an ETA for Cloudformation support of EFS for Fargate? We are eagerly waiting for that feature

@uherberg we are actively working on it. I don't have more details to share at this time. Stay tuned. We will update this post when CloudFormation support is introduced. Thanks for your patient.

Has CloudFormation support been introduced? I'm thinking about including the EFS resource in the template to be able to reference its id in the TaskDefinition

[diraven]

Yes, it was. I was able to successfully use it. ср, 3 лют. 2021, 13:22 користувач Stefano <[email protected]> пише:

…

@mreferre <https://github.com/mreferre> Is there an ETA for Cloudformation support of EFS for Fargate? We are eagerly waiting for that feature @uherberg <https://github.com/uherberg> we are actively working on it. I don't have more details to share at this time. Stay tuned. We will update this post when CloudFormation support is introduced. Thanks for your patient. Has CloudFormation support been introduced? I'm thinking about including the EFS resource in the template to be able to reference its id in the TaskDefinition — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#53 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA2N5ZSU35GCYGWPRBJGS3S5EWXNANCNFSM4GKCYBVQ> .

[mreferre]

Yes. I apologize, I committed to update the thread when we announced support and I didn't. Sorry.

Here for the official announcement: https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-ecs-announces-cloudformation-support-for-amazon-efs-volumes/

[babaMar]

No problem at all @mreferre ! Attempting right now, hopefully as successful as @diraven

Getting container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:71: creating device nodes caused: errno 524 actually

Here's the relevant parts of the template:

  TaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          Effect: Allow
          Principal:
            Service: ecs-tasks.amazonaws.com
          Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
        - PolicyName: EFSPermissions
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - elasticfilesystem:*
                Resource:
                  - !GetAtt DataAccessPoint.Arn
                  - !GetAtt DataVolume.Arn

  ServiceTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ContainerDefinitions:
        - Environment:
            - Name: SERVER_DOMAIN
              Value: !Ref DomainName

          Essential: true
          Image: !Sub '${RepositoryURL}:${CommitHash}'
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref LogGroup
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: task
          PortMappings:
            - ContainerPort: 80
          Privileged: false
          MountPoints:
            - ContainerPath: '/'
              SourceVolume: 'storage'

      Cpu: '1024'
      Memory: '4096'
      NetworkMode: awsvpc
      ExecutionRoleArn: !GetAtt TaskExecutionRole.Arn
      TaskRoleArn: !GetAtt TaskExecutionRole.Arn
      RequiresCompatibilities: [FARGATE]
      Volumes:
        - Name: 'storage'
          EFSVolumeConfiguration:
            AuthorizationConfig:
              AccessPointId: !GetAtt DataAccessPoint.AccessPointId
              IAM: 'ENABLED'
            FileSystemId: !Ref DataVolume
            TransitEncryption: 'ENABLED'

  DataVolume:
    Type: AWS::EFS::FileSystem
    Properties:
      BackupPolicy:
        Status: ENABLED
      Encrypted: false
      PerformanceMode: 'maxIO'
      ThroughputMode: 'bursting'

  DataMountTarget:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref DataVolume
      SubnetId: !ImportValue InternetEdgeInternalSubnetId
      SecurityGroups:
      - !ImportValue AllowInternalAddressesSecurityGroup

  DataAccessPoint:
    Type: AWS::EFS::AccessPoint
    Properties:
      AccessPointTags:
        - Key: 'scope'
          Value: 'jupyterhub'
      FileSystemId: !Ref DataVolume

[mreferre]

I don't have a working example handy so it's hard for me to see what could be wrong. Do you have the other parts of the CFN template out on purpose? (MountTarget's etc)
Also, where are you getting that error? Is that what the container spit on stdout and that you are capturing in the container log? Are you sure that error message maps to a failure in mounting the EFS volume? If you are deploying the task on EC2 can you exec into the container and inspect if the volume has been mounted? Sorry for the generic questions but I am trying to narrow the scope of the potential causes.

[babaMar]

Thanks for you reply. Included the AWS::EFS::MountTarget section in the comment above.

The error shows up in the stdout of the Task within the ECS Service, I'm not deploying in EC2. I'm pretty sure the failure is due to the volume mounting, as it works if I disable the volume option (and on local).

Do you maybe have a working example of a template?

[mreferre]

There are other parts to check (such as the security groups). I am not great at debugging CFN templates this way. I have a few suggestions that may put you on the right track.

• this is a CLI walkthrough to set up a simple task that connects to an EFS share. You could go through it and compare the differences of what gets deployed/configured with these commands Vs your CFN template.
• I have just used the docker compose integration described here to "convert" a simple docker compose file into an equivalent (working) CFN template that spins up an ECS service whose task connect to the EFS Access Point. This is the docker-compose.yaml I have used as an input:

version: "3.8"
services:
  nginx:
    image: nginx
    volumes:
      - mydata:/tmp
    ports: 
      - 80:80
volumes:
  mydata:

To generate the corresponding CFN template from the compose file you just run docker compose convert. If you don't want to go through the setup of the docker compose CLI as described in the blog contact me off line and I will share my CFN template (not posting it here because it contains explicit resources in my account).