endpoint

AWS EC2 Spot Instance Termination Simulator

by
featured image for aws ec2 spot instance termination simulator blog post

In this post I’ll explain how I created an AWS EC2 Spot Instance Termination Simulator to run on any EC2 instance.

It can signal typical monitoring scripts, applications, interrupt handlers, and more, just like a real Spot Termination signal would do.

Back story

Looking around, there aren’t really any straight forward ways of testing this behaviour.

The obvious way is to change the price you bid for spot instances to the threshold of the current market price. This is so that if there is a slight increase in demand your spot instance(s) will be terminated.

The problem with that approach is of course that you can’t easily predict when the price will move, or by how much.

How EC2 Spot Instance Termination warnings work

When your spot instance bid price is surpassed by the current market price, a Termination Notice becomes available on one of the instance’s metadata endpoints. For example, http://169.254.169.254/latest/meta-data/spot/termination-time.

There is one other, newer endpoint at http://169.254.169.254/latest/meta-data/spot/instance-action that could also be used.

At this point the endpoint returns an HTTP 200 response. A timestamp of when a shutdown signal is going to be sent to your instance’s OS is also returned.

In the case of the newer endpoint, a JSON string is returned with an action and time. Usually the endpoint usually simply returns a 404 not found response.

The tools out there tend to monitor this endpoint to warn and action in case of a Spot Termination notice being received.

The two minute warning feature was added by Amazon back in 2015, announced in this blog post.

Kubernetes Spot Termination / Interrupt handling

I’ve been looking at kube-aws/kube-spot-termination-notice-handler.

It runs as a DaemonSet (one container/pod on each node) in host networking mode. It then polls the EC2 metadata endpoint every ‘x’ number of seconds, watching for termination signals.

During polling, if one is found, it will send a kubectl drain command to it’s host node. Pods will move off to other nodes in the cluster.

Simulating Spot Instance Termination with a fake web service and some proxying

I created a simple web service/API that simply returns a 200 HTTP response on the same http://169.254.169.254/latest/meta-data/spot/termination-time endpoint.

This is the legacy endpoint that spot instance termination signals go to. However, there is another, newer one that AWS now recommend using instead.

The other endpoint is http://169.254.169.254/latest/meta-data/spot/instance-action. The web service will also send a response there as a JSON string with an action and time field.

All that needs to be done is to forward traffic from the metadata endpoint 169.254.169.254:80 to the custom web service.

Have a look at the code and Docker image at the following locations:

Run the EC2 spot instance termination simulator endpoints

Identify a candidate EC2 instance that you don’t mind messing with.

Warning: the following steps will override the entire EC2 instance metadata service. No other metadata endpoints will work. This is because they’re not implemented in this web service.

Kubernetes

Deploy the docker container (a Kubernetes deployment / service manifest is in the git repository).

Ideally, create a nodeSelector / label up in the deployment manifest before you kubectl apply it. This is so you can get the pod to run on the specific node you want to test.

kubectl -n namespace apply -f https://raw.githubusercontent.com/Shogan/ec2-spot-termination-simulator/master/simple-k8s-deployment.yaml

The service is a NodePort service, exposing the port on the host it runs on.

List the service and the pod (and find the kubernetes node the pod is running on):

kubectl get svc ec2-spot-termination-simulator
kubectl get pod spot-term-simulator-xxxxxx -o wide

Take note of the NodePort the service is listening on for the Kubernetes node. E.g. port 30626.

Docker

docker run -p 30626:80 -e PORT=80 shoganator/ec2-spot-termination-simulator

Proxying the EC2 metadata service

Perform some trickery to proxy traffic destined to 169.254.169.254 on port 80 to localhost (where the container runs). This is so that the fake service can take the place of the real one.

SSH onto the Kubernetes node and run:

sudo ifconfig lo:0 169.254.169.254 up
sudo socat TCP4-LISTEN:80,fork TCP4:127.0.0.1:30626
  1. Create an alias for the localhost interface at 169.254.169.254, effectively taking over the EC2 metadata service and sending that to 127.0.0.1 instead.
  2. Forward TCP port 80 traffic (usually destined to the EC2 metadata service) to 127.0.0.1 on the NodePort 30626. This is where the ec2-spot-termination-simulator pod is running on this host. Substitute this with the correct port in your case.

Test the new, faked endpoint:

curl -i http://169.254.169.254/latest/meta-data/spot/termination-time

As a result, it should return a 200 OK response with a timestamp from the fake service. This is the same as the real one would.

Example

Looking at how the kube-spot-termination-notice-handler service works specifically:

This service runs as a DaemonSet, meaning one instance per host. The instance on the node you set the simualtor up on should immediately drain the node. The instance won’t actually be terminated as this was of course just a simulated termination.

Other scenarios

If you’re not running on Kubernetes and are using a different spot termination handler, don’t worry. The system you’re using to monitor the EC2 instance metadata endpoint should still take action at this point.

The proxied web service is now returning a legitimate looking termination time notice on http://169.254.169.254/latest/meta-data/spot/termination-time.

Octopus Deploy Endpoint auto configuration on Azure VM deployment

by

I’ve been working on a very cool project that involves the use of Microsoft Azure, TeamCity and Octopus Deploy.

I have created an Azure PowerShell script that deploys VMs into an Azure Subscription (Web machines that run IIS) as a part of a single Azure Cloud Service with load balancing enabled. As such, the endpoint ports that I create for Octopus tentacle communication need to differ for each machine on the public interface.

I wanted to fully automate things from end-to-end, so I wrote a very small console application that uses the Octopus Client library NuGet package in order to be able to communicate with your Octopus Deploy server via the HTTP API.

Octopus Endpoint Configurator on GitHub

The OctopusConfigurator console application should be run in your Azure VM once it is deployed, with 4 x parameters to specify when run.

It will then establish communication with your Octopus Deploy server, and register a new Tentacle endpoint using the details you pass it. The standard port number that gets assigned (10933) will then be replaced if necessary with the correct endpoint port number for that particular VM instance in your cloud service. For example, I usually start the first VM in my cloud service off on 10933, then increment the port number by 1 for every extra VM in the cloud service. As the deployments happen, the console application registers each new machine’s tentacle using the incremented port number back with the Octopus master server.

Once the Azure VM deployment is complete, I tell the VMs in the cloud service to restart with a bit of Azure PowerShell and once this is done, your Octopus environment page should show all newly deployed tentacles as online for your environment. Here is an example of an Invoke-Command scriptblock that I execute remotely on my Azure VMs as soon as they have completed initial deployment. What I do is tell the VM deployment script to wait for Windows boot, so once ready, the WinRM details are fetched for the VM using the Get-AzureWinRMUri cmdlet for Azure, which allows me to use the Invoke-Command to run the below script inside the guest VM.

 

Invoke-Command -ConnectionUri $connectionString -Credential $creds -ArgumentList $vmname,$externalDNSName,$creds,$InstallTentacleFunction,$OctopusExternalPort,$OctopusEnvironmentName -ScriptBlock {
	
	$webServerName = $args[0]
    $DNSPassthrough = $args[1]
    $passedCredentials = $args[2]
    $scriptFunction = $args[3]
    $OctoPort = $args[4]
    $OctopusEnvironmentName = $args[5]
		
	function DownloadFileUrl($url, $destinationPath, $fileNameToSave)
	{
	    $fullPath = "$destinationPath\$fileNameToSave"

	    if (Test-Path -Path $destinationPath)
	    {
	        Invoke-WebRequest $url -OutFile $fullPath
	    }
	    else
	    {
	        mkdir $destinationPath
	        Invoke-WebRequest $url -OutFile $fullPath
	    }

	    Write-Host "Full path is: $fullPath"
	    return [string]$fullPath
	}
	
	# Download the Octopus Endpoint Configurator to C:\Temp
	[string]$ConfiguratorPath = DownloadFileUrl "https://dl.dropboxusercontent.com/u/xxxxxxx/Apps/OctopusConfigurator.zip" "C:\Temp" "OctopusConfigurator.zip"
	
	Write-Host "Unzipping OctopusConfigurator.zip" -ForegroundColor Green
    cd C:\Temp
    $shell_app=new-object -com shell.application
    $filename = "OctopusConfigurator.zip"
    $zip_file = $shell_app.namespace((Get-Location).Path + "\$filename")
    $destination = $shell_app.namespace((Get-Location).Path)
    $destination.Copyhere($zip_file.items())
	
    cd C:\Temp

    if (Test-Path -Path .\OctopusConfigurator.exe)
    {
        & .\OctopusConfigurator.exe http://theoctopusurl.domain API-XXXXXXXXXXXXXXXXXXXXXX $webServerName $OctoPort
        Write-Host "Reconfigured Octopus Machine URI to correct port number" -ForegroundColor Green
    }
    else
    {
        Write-Host "OctopusConfigurator not found!" -ForegroundColor Red
        Exit
    }
}