EFS in an ECS Cluster

Creating an Elastic File System for an ECS cluster is a useful tool for sharing data between ECS tasks (Docker containers).

There are several useful use cases, for example:

• Content management applications, for performance and redundancy
• Code and artifacts persisted to multiple AWS availability zones, for durability.
• Machine learning frameworks access to data, for having a durable shared storage.
• Shared notebook systems like Jupyter, for durable & shared storage.

For more information, see best usage practices here.

EFS can be super secured too, not only with network security (security groups, private subnets, etc) but also using authorization and application security with the new featured announced in 2020.

Just like people, containers can share resources

Setting Up EFS using Terraform:

First, create the EFS and mount target resources:

//efs.tf
resource "aws_efs_file_system" "efs" {
  creation_token = "vizefs"
  encrypted = true

  tags = {
    Name = "${var.base_efs_prefix}efs"
  }
}

resource "aws_efs_mount_target" "mount" {
  file_system_id = "${aws_efs_file_system.efs.id}"
  subnet_id = "${element("${var.efs_subnets}", count.index)}"
  security_groups = "${var.efs_security_group}"
  count = "${length("${var.efs_subnets}")}"
}

and the output file:

//output.tf
output "efs_dns_name" {
  value = "${aws_efs_file_system.efs.dns_name}"
}

Notice it’s best practice to have the file system available in a few subnets for durability. We’ll use the Terraform element function in order to achieve that:

subnet_id = "${element("${var.efs_subnets}", count.index)}"

Now that we have our resources, we’ll want to mount the EFS in the EC2s running our ECS tasks. For that we’ll use the use data script:

mkdir -p /opt/efs-shared-volume
cd /opt/efs-shared-volume
sudo chmod +XXX /opt/efs-shared-volume
apt-get -y install nfs-common
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport ${efs_dns_name}:/ .

Notice also how we’re utilizing our efs_dns_name from the output file.

At this point we can run our TF code and have the EFS available across all of the EC2 instances (which are running the cluster’s tasks). The next step would be to create a mount between the ECS task and the EC2 host using bind mounts. We’ll go into that in a following post.