Creating an Elastic File System for an ECS cluster is a useful tool for sharing data between ECS tasks (Docker containers).

There are several useful use cases, for example:

  • Code and artifacts persisted to multiple AWS availability zones, for durability.
  • Machine learning frameworks access to data, for having a durable shared storage.
  • Shared notebook systems like Jupyter, for durable & shared storage.

For more information, see best usage practices here.

EFS can be super secured too, not only with network security (security groups, private subnets, etc) but also using authorization and application security…


Building and sharing experiments quickly is an important requirement when working with a global team consisting of different technical capabilities and expertise.

There are various tools for both running and sharing ML/DL research work. In this post I’ll review the process of creating and sharing a DL model using Corlab & Binder.

Corlab, a tool created by Google that allows spinning off a free Jupiter notebook which can run on a CPU or a GPU. No need for a strong local machine nor a local setup, it just works smoothly with a web browser.


Scheduled ASG actions can be used for a variety of reasons; one typical is to handle predicted rush hours at certain times of the days, special dates etc.

This is a well known feature in EC2, however in ECS it’s relatively hidden, available via the CLI only; I assume the team will work on it at some point — if you find it useful for you own needs too — please vote here!

Create an ECS Scheduled Action via CLI:

aws application-autoscaling put-scheduled-action --service-namespace ecs --scalable-dimension ecs:service:DesiredCount --resource-id service/<cluster-name>/<service-name> --scheduled-action-name some_name --scalable-target-action MinCapacity=3 --schedule "cron(59 * * * ? …


A Task Self-destruct Mechanism For ECS

In the last post I have demonstrated that ECS might terminate our tasks if we don’t build the appropriate infrastructure. In this post I’ll attempt to build a successful mechanism that works as following:

  • The ECS agent sends a ‘SIGTERMsignal to the running docker container
  • The signal is being caught by the application running inside the docker container.
  • The application checks if it is currently processing, if not exits
  • If it is processing, it waits until processing is…

In order to run a processing pipeline / batch (e.g a set of tasks that process for a finite amount of time) in ECS one has two main options: use an ECS service or initiate a standalone task.

An ECS service will probably listen to a queue, a metric or an API call that will trigger the service which will then start processing. …


Photo by Christopher Gower on Unsplash

To run a GPU based task in ECS, we need to create our own EC2 instance as Fargate still doesn’t support GPUs. That shouldn’t be too hard with the ECS GPU optimized AMIs.

However, sometimes that cannot be done as teams are already using their favorite AMI setups such as an Ubuntu CIS optimized AMI or any other flavor. This means they need to install and configure the setup from scratch.

In this set of 4 articles, we’ll review the installation and configuration process of an ECS task with GPU required resources over an Ubuntu 18.04 OS.

Part 1: The…


Photo by Daniel Korpai on Unsplash

In order to run a GPU based task in ECS, we need to create our own EC2 instance as Fargate still doesn’t support GPUs. That shouldn’t be too hard with the ECS GPU optimized AMIs.

However, sometimes that cannot be done as teams are already using their favorite AMI setups such as an Ubuntu CIS optimized AMI or any other flavor. This means they need to install and configure the setup from scratch.

In this set of 4 articles, we’ll review the installation and configuration process of an ECS task with GPU required resources over an Ubuntu 18.04 OS.

Part…


Photo by Oscar Nilsson on Unsplash

In order to run a GPU based task in ECS, we need to create our own EC2 instance as Fargate still doesn’t support GPUs. That shouldn’t be too hard with the ECS GPU optimized AMIs.

However, sometimes that cannot be done as teams are already using their favorite AMI setups such as an Ubuntu CIS optimized AMI or any other flavor. This means they need to install and configure the setup from scratch.

In this set of 4 articles, we’ll review the installation and configuration process of an ECS task with GPU required resources over an Ubuntu 18.04 OS.

Part…


In order to run a GPU based task in ECS, we need to create our own EC2 instance as Fargate still doesn’t support GPUs. That shouldn’t be too hard with the ECS GPU optimized AMIs.

However, sometimes that cannot be done as teams are already using their favorite AMI setups such as an Ubuntu CIS optimized AMI or any other flavor. This means they need to install and configure the setup from scratch.

In this set of 4 articles, we’ll review the installation and configuration process of an ECS task with GPU required resources over an Ubuntu 18.04 OS.

Part…


The new delivery standard ?

Waypoint is a new software delivery tool made by HashiCorp, released last week. It’s a new abstraction layer over the build-deploy-release pipeline which provides an ‘infrastructure as code’ like approach to the entire delivery flow.

In this post we’ll cover a couple of introductory operations:

  • Push the docker image into a Docker registry

Michael Loewenstein

👨🏻‍💻 Engineering Leader ⛰️ Software Developer ☁️ Cloud Solution Architect

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store