Building and sharing experiments quickly is an important requirement when working with a global team consisting of different technical capabilities and expertise.

There are various tools for both running and sharing ML/DL research work. In this post I’ll review the process of creating and sharing a DL model using Corlab & Binder.

Corlab, a tool created by Google that allows spinning off a free Jupiter notebook which can run on a CPU or a GPU. No need for a strong local machine nor a local setup, it just works smoothly with a web browser.

Scheduled ASG actions can be used for a variety of reasons; one typical is to handle predicted rush hours at certain times of the days, special dates etc.

This is a well known feature in EC2, however in ECS it’s relatively hidden, available via the CLI only; I assume the team will work on it at some point — if you find it useful for you own needs too — please vote here!

Create an ECS Scheduled Action via CLI:

aws application-autoscaling put-scheduled-action --service-namespace ecs --scalable-dimension ecs:service:DesiredCount --resource-id service/<cluster-name>/<service-name> --scheduled-action-name some_name --scalable-target-action MinCapacity=3 --schedule "cron(59 * * * ? …

A Task Self-destruct Mechanism For ECS

In the last post I have demonstrated that ECS might terminate our tasks if we don’t build the appropriate infrastructure. In this post I’ll attempt to build a successful mechanism that works as following:

• ECS sends a ‘terminate task’ signal to the running ECS task (due to a scale-in/deployment).
• The ECS agent sends a ‘SIGTERM’ signal to the running docker container
• The signal is being caught by the application running inside the docker container.
• The application checks if it is currently processing, if not exits
• If it is processing, it waits until processing is…

In order to run a processing pipeline / batch (e.g a set of tasks that process for a finite amount of time) in ECS one has two main options: use an ECS service or initiate a standalone task.

An ECS service will probably listen to a queue, a metric or an API call that will trigger the service which will then start processing. …

To run a GPU based task in ECS, we need to create our own EC2 instance as Fargate still doesn’t support GPUs. That shouldn’t be too hard with the ECS GPU optimized AMIs.

However, sometimes that cannot be done as teams are already using their favorite AMI setups such as an Ubuntu CIS optimized AMI or any other flavor. This means they need to install and configure the setup from scratch.

In this set of 4 articles, we’ll review the installation and configuration process of an ECS task with GPU required resources over an Ubuntu 18.04 OS.

Part 1: The…

In order to run a GPU based task in ECS, we need to create our own EC2 instance as Fargate still doesn’t support GPUs. That shouldn’t be too hard with the ECS GPU optimized AMIs.

However, sometimes that cannot be done as teams are already using their favorite AMI setups such as an Ubuntu CIS optimized AMI or any other flavor. This means they need to install and configure the setup from scratch.

In this set of 4 articles, we’ll review the installation and configuration process of an ECS task with GPU required resources over an Ubuntu 18.04 OS.

Part…

In order to run a GPU based task in ECS, we need to create our own EC2 instance as Fargate still doesn’t support GPUs. That shouldn’t be too hard with the ECS GPU optimized AMIs.

However, sometimes that cannot be done as teams are already using their favorite AMI setups such as an Ubuntu CIS optimized AMI or any other flavor. This means they need to install and configure the setup from scratch.

In this set of 4 articles, we’ll review the installation and configuration process of an ECS task with GPU required resources over an Ubuntu 18.04 OS.

Part…

In order to run a GPU based task in ECS, we need to create our own EC2 instance as Fargate still doesn’t support GPUs. That shouldn’t be too hard with the ECS GPU optimized AMIs.

However, sometimes that cannot be done as teams are already using their favorite AMI setups such as an Ubuntu CIS optimized AMI or any other flavor. This means they need to install and configure the setup from scratch.

In this set of 4 articles, we’ll review the installation and configuration process of an ECS task with GPU required resources over an Ubuntu 18.04 OS.

Part…