AWS re-attach EBS volume on instance creation

What:

Amazon Web Services - EC2, ECS, EBS

Problem:

You have some data on EBS volume which you don't want to destroy when instance is terminated.

Solution:

I use spot instances in AWS for cost saving purposes. Spot instances (https://aws.amazon.com/ec2/spot/) are terminated and re-launched when price fluctuate, so I needed a way of keeping data while instances are being re-created. It's worth mentioning that in my case my persistent data is for elasticsearch which runs on ECS.

Follow below steps to attach EBS volume on instance creation and add them to the ECS cluster.
First create spot instance request, navigate to EC2 > Instances > Spot Requests
Click on "Request Spot Instances"
Choose your desired settings and click Next
Under "Set instance details" you will find "User data" field
This is where you can insert a script which will be executed on instance creation.

Insert following code:

#!/bin/bash
# userdata script for ECS cluster, which will attach existing and available encrypted elb volume called "ecs-cluster"
# if volume with above criteria can't be found it will create a new one

yum install -y aws-cli
# Use access key for testing only, you should use IAM role and proper policy instead
export AWS_ACCESS_KEY_ID=<your_id>
export AWS_SECRET_ACCESS_KEY=<secret_key>

aws_region="us-east-1"
ebs_name="ecs-cluster"
ebs_size="100"
inst_id=$(curl http://169.254.169.254/latest/meta-data/instance-id)
av_zone=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)

discover_aval_vol() {
    ebs_av_vol=$(aws ec2 describe-volumes --region ${aws_region} --filters "Name=status,Values=available" "Name=tag:Name,Values=${ebs_name}" "Name=availability-zone,Values=${av_zone}" | awk '/VolumeId/ {gsub(",","");gsub("\"",""); print $2}' | head -1)
}
attach_volume() {
    aws ec2 attach-volume --region ${aws_region} --volume-id ${ebs_av_vol} --instance-id ${inst_id} --device /dev/sdb
    attv_retcode=$?
    # wait for the volume to be attached before we mount it
    while [ ! -e /dev/sdb ]; do sleep 2; done
}
create_volume() {
    echo "No available ebs volumes found, creating new one..."
    aws ec2 create-volume --tag-specifications "ResourceType=volume,Tags={Key=Name,Value=${ebs_name}}" --encrypted --size ${ebs_size} --region ${aws_region} --availability-zone ${av_zone} --volume-type gp2
    sleep 60
}
discover_attach_or_create() {
    echo "Discovering available volumes..."
    discover_aval_vol
    echo "Found following volume: ${ebs_av_vol}"
    if [ -n "${ebs_av_vol}" ]; then
        echo "Attaching available volume: ${ebs_av_vol}"
        attach_volume
        if [ ${attv_retcode} -ne 0 ]; then
            echo "Error: Volume already attached to a different instance"
            success=1
            return
        fi
    else
        create_volume
        success=2
        return
    fi
    success=0
}
# tag instance
aws ec2 --region ${aws_region} create-tags --resources ${inst_id} --tags Key=Name,Value="EC2 ECS"

# sleep from 0 to 60s to avoid requesting volumes at the same time
let value=${RANDOM}%60
sleep ${value}
success=5
n=0
while [ ${success} -ne 0 ]; do
    discover_attach_or_create
    n=$[${n}+1]
    if [ ${n} -ge 2 ]; then
        break
    fi
done

echo ECS_CLUSTER=ecs-cluster >> /etc/ecs/ecs.config
echo vm.max_map_count=262144 >> /etc/sysctl.conf
sysctl -w vm.max_map_count=262144
echo vm.swappiness=1 >> /etc/sysctl.conf
sysctl -w vm.swappiness=1
sed -i 's/\(default-ulimit nofile=[0-9]\+\:\)[0-9]\+/\165536/' /etc/sysconfig/docker

# create filesystem, but only for new volumes
file -Ls /dev/sdb | grep ext4 || mkfs -t ext4 /dev/sdb
mkdir /opt/elasticsearchdata
mount /dev/sdb /opt/elasticsearchdata
echo "/dev/sdb /opt/elasticsearchdata ext4 defaults,nofail 0 2" >> /etc/fstab
chown -R 991:991 /opt/elasticsearchdata
service docker restart
start ecs

Click Review, if you're happy with the settings you can now download configuration in JSON format.
Click Launch to start launching your spot instances.

User data script is also part of Launch Configuration, so it can be used to configure any instance not only "spots".