Octopus Deploy Endpoint auto configuration on Azure VM deployment

I’ve been working on a very cool project that involves the use of Microsoft Azure, TeamCity and Octopus Deploy.

I have created an Azure PowerShell script that deploys VMs into an Azure Subscription (Web machines that run IIS) as a part of a single Azure Cloud Service with load balancing enabled. As such, the endpoint ports that I create for Octopus tentacle communication need to differ for each machine on the public interface.

I wanted to fully automate things from end-to-end, so I wrote a very small console application that uses the Octopus Client library NuGet package in order to be able to communicate with your Octopus Deploy server via the HTTP API.

Octopus Endpoint Configurator on GitHub

The OctopusConfigurator console application should be run in your Azure VM once it is deployed, with 4 x parameters to specify when run.

It will then establish communication with your Octopus Deploy server, and register a new Tentacle endpoint using the details you pass it. The standard port number that gets assigned (10933) will then be replaced if necessary with the correct endpoint port number for that particular VM instance in your cloud service. For example, I usually start the first VM in my cloud service off on 10933, then increment the port number by 1 for every extra VM in the cloud service. As the deployments happen, the console application registers each new machine’s tentacle using the incremented port number back with the Octopus master server.

Once the Azure VM deployment is complete, I tell the VMs in the cloud service to restart with a bit of Azure PowerShell and once this is done, your Octopus environment page should show all newly deployed tentacles as online for your environment. Here is an example of an Invoke-Command scriptblock that I execute remotely on my Azure VMs as soon as they have completed initial deployment. What I do is tell the VM deployment script to wait for Windows boot, so once ready, the WinRM details are fetched for the VM using the Get-AzureWinRMUri cmdlet for Azure, which allows me to use the Invoke-Command to run the below script inside the guest VM.

 

Invoke-Command -ConnectionUri $connectionString -Credential $creds -ArgumentList $vmname,$externalDNSName,$creds,$InstallTentacleFunction,$OctopusExternalPort,$OctopusEnvironmentName -ScriptBlock {
	
	$webServerName = $args[0]
    $DNSPassthrough = $args[1]
    $passedCredentials = $args[2]
    $scriptFunction = $args[3]
    $OctoPort = $args[4]
    $OctopusEnvironmentName = $args[5]
		
	function DownloadFileUrl($url, $destinationPath, $fileNameToSave)
	{
	    $fullPath = "$destinationPath\$fileNameToSave"

	    if (Test-Path -Path $destinationPath)
	    {
	        Invoke-WebRequest $url -OutFile $fullPath
	    }
	    else
	    {
	        mkdir $destinationPath
	        Invoke-WebRequest $url -OutFile $fullPath
	    }

	    Write-Host "Full path is: $fullPath"
	    return [string]$fullPath
	}
	
	# Download the Octopus Endpoint Configurator to C:\Temp
	[string]$ConfiguratorPath = DownloadFileUrl "https://dl.dropboxusercontent.com/u/xxxxxxx/Apps/OctopusConfigurator.zip" "C:\Temp" "OctopusConfigurator.zip"
	
	Write-Host "Unzipping OctopusConfigurator.zip" -ForegroundColor Green
    cd C:\Temp
    $shell_app=new-object -com shell.application
    $filename = "OctopusConfigurator.zip"
    $zip_file = $shell_app.namespace((Get-Location).Path + "\$filename")
    $destination = $shell_app.namespace((Get-Location).Path)
    $destination.Copyhere($zip_file.items())
	
    cd C:\Temp

    if (Test-Path -Path .\OctopusConfigurator.exe)
    {
        & .\OctopusConfigurator.exe http://theoctopusurl.domain API-XXXXXXXXXXXXXXXXXXXXXX $webServerName $OctoPort
        Write-Host "Reconfigured Octopus Machine URI to correct port number" -ForegroundColor Green
    }
    else
    {
        Write-Host "OctopusConfigurator not found!" -ForegroundColor Red
        Exit
    }
}

VM provisioning from e-mail using Python and the VMware Perl SDK

This is a bit of a fun project that I did as a part of my presentation of the vPi project. It doesn’t necessarily achieve anything useful (at least not on the surface), but it does demonstrate some techniques that could be put to far greater use.

 

vpi-provision-script

 

In summary, this integration turns e-mails from people into Virtual Machines on a vSphere environment. It consists of the following components:

  • Raspberry Pi running the vPi image
  • Python script
  • VMware Perl script (vmcreate.pl) + a bit of XML used for the VM template.
  • VMware Perl script (vmcontrol.pl)

The way it works, is a Gmail mailbox is setup to capture e-mails sent to a specific e-mail address. The Raspberry Pi runs a Python script that logs into Gmail, and looks for any new e-mail that has arrived. If an e-mail is found, it takes the FROM address and splits it up into components, determining the sender’s first and last names.

The script then opens up the XML template file that the vmcreate.pl script uses as a basis to creating VMs, and searches it for a bit of bespoke text we placed there called “TEMPLATE_NAME”. Once found, it strips this out and replaces the TEMPLATE_NAME with the sender’s name.

We then move onto the next procedure, which involves invoking the vmcreate.pl script from the Python script, passing it in the parameters required (such as the server to connect to, credentials, and the all important XML template). This runs against the vSphere environment in question, and creates a VM named after the e-mail sender, (appending a random string of text and numbers to the end to ensure that multiple e-mails from the same person do not cause an issue with duplicate named VMs).

Once the VM is provisioned, the Python script invokes the vmcontrol.pl script using the name of the VM we just provisioned to power the VM up. Lastly, the Python script sends an e-mail back to the sender, stating that their VM has been created and powered on. After that, voila! You will have a new VM created and deployed in your Datacenter all triggered from a simple e-mail.

vpi-example

The script files required + XML and XML schema files are available for download below. The main python script is fairly lengthy, so I won’t include the content direct on this post. Just download the file to grab everything.

Notes to get the script up and running:

  • I found the vmcreate.xsd (XML Schema file) for the VMware vmcreate.pl script did not work, so I had to modify it to change some of the property names to match those of which the vmcreate script was expecting. My updated version is included in the download below if you get any errors from the vmcreate.pl script. It’s default location is: /usr/lib/vmware-viperl/apps/schema
  • You will need to find and edit some variables in the main Python script – your mailbox name and password, plus the IP, username and password for the vmcreate.pl and vmcontrol.pl perl script calls.
  • In the vmtemplate.xml file you should define the characteristics of your VMs that are created. GuestOS, Disk size, etc… Of particular importance, is the name of your host to deploy to, Datacenter name, Datastore name to deploy to, and default VM network to use. These are all of course unique to your own environments.

 

[download id=”26″ template=””]
 
[download id=”27″ template=””]

 

Once you start to think of other ways of using this, you can begin to imagine some really great (and even crazy) solutions. As a start, it would be quite easy to begin extending this, so that e-mails undergo some sort of validation first. E.g. does the domain the sender sent from exist in our “Whitelist” of people allowed to provision e-mails, or does a specific “password” required exist in the body of the e-mail etc…

How about having a standard e-mail template, where the sender can specify more details, such as vCPUs, RAM, disk sizes, OS to install? You could then provision from VM templates instead of creating new VMs, that have customization specs attached. Once powered up and provisioned, a script within the VM could be initiated to accept parameters the VM was created with, and use those to send the requestor an e-mail to say “Hey! I’m now ready for you to connect, and here is the IP you can use…”.

 

Of course, this is not limited to vPi and the Raspberry Pi – that was just the platform I demonstrated this on. Being standard SDKs and scripting languages, you could use the above solution anywhere.

 

vPi – a Raspberry Pi for VMware presentation on vBrownBag EMEA

Last week I hosted a session talking about vPi. vPi is an image for the Raspberry Pi based on Raspbian, specifically targeted at VMware integration. Many great things can be done with this solution, including some very nifty automation, scripting and reporting.

The session was hosted on the weekly vBrownBag EMEA webinar. Take a look at the embedded Vimeo video below to watch through. Here is a basic breakdown of what the session covers:

  • Introduction / about
  • Basic intro to Raspberry Pi
  • Basic intro to vPi
  • Live demo
    • Demonstration of various VMware utilities included with vPi, such as ESXCLI, the RVC fling, vmkfstools, etc…
    • Very cool provisioning script demo using Python and Perl to deploy Virtual Machines by users sending a simple e-mail through to a designated mailbox (audience participation)
    • Quick demo of a home automation script that integrates with Foursquare, Facebook, and local Weather channels to announce various statuses/states
  • Conclusion

Remember to switch to HD mode so you can read the text in the presentation and PuTTy session I had open. Also a big thanks to Gregg Robertson (co-worker and fellow saffa) for inviting me to present on last week’s vBrownBag EMEA session.

 

httpvh://vimeo.com/71875957

 

Slides are available for download here, but it probably makes more sense to check the whole presentation out on Vimeo, as the demo is the bulk of the session:

[download id=”26″]

How to use PoSH or PowerCLI to SSH into network devices – wodSSH [Part 2]

 

In [part 1] of this series of posts, I explored SharpSSH, and how to use this to SSH into network devices via PowerShell or PowerCLI. I found this extremely useful, and managed to automate a process of retrieving some SHA1 SSL fingerprints from a few clusters of ESX hosts in the process. I did however find that loading the dll and functions necessary to get this bit of automation done was a little tedious. I had heard about wodSSH via the get-scripting podcast and decided I would need to take a look at this.

 

If you are planning on using wodSSH, you should start by downloading it here. Note that you should definitely buy a license if you are going to be using this for your job or day to day automation – the license is not to expensive if you consider the kinds of automation you could achieve with this! For this blog post I am just using the trial version in my lab environment at home to try out – the trial pops up a “nag screen” every now and then to indicate you are unlicensed. Once downloaded, install wodSSH and the libraries will now be available for use on your machine.

 

Download the script below, or take a look at it in the Syntax Highlighted area below – this is an example of how I was able to SSH into a cluster of ESXi hosts and run a remote command via SSH through PowerCLI. In this example, a table is returned with Host names, along with their SHA1 fingerprint information. It is possible to retrieve this info (as of ESXi 5 I believe) in PowerCLI, but for this example I just wanted to demonstrate the use of SSH and PowerShell / PowerCLI.

 

[download id=”14″]

 

$login = "root"
$password = "yourrootpassword" 

$Report = @()
$VMHosts = Get-VMHost | Where {$_.ConnectionState -eq "Connected"} | Sort Name

foreach ($vmhost in $VMHosts) {
	$hostname = $vmhost
	$ssh = New-Object -ComObject "weOnlyDo.wodSSHCom.1"
	$ssh.Hostname = $hostname
	$ssh.Login = $login
	$ssh.Password = $password
	$ssh.Blocking = 1
	$ssh.Protocol = 4
	$ssh.Timeout = 25
	$ssh.Connect($hostname)
	$ssh.WaitFor("regex:[$%#>] $")
	$cmdresult = $ssh.Execute("openssl x509 -sha1 -in /etc/vmware/ssl/rui.crt -noout -fingerprint`r`n","regex:[~#]")
	$ssh.Disconnect()
	$temp = $cmdresult | select-string -pattern "SHA1 Fingerprint="
	Write-Host $cmdresult -ForegroundColor Green
	$row = New-Object -TypeName PSObject -Property @{
		SHA1 = $cmdresult
		HostName = $vmhost
	}
	$Report += $row
}
$Report

 

You will notice that on line 09, we load the wodSSH library using the New-Object cmdlet. After this we continue to setup some properties for our $ssh connection object. A regex check is used to determine when are connected to a host and have an SSH shell prompt ($ssh.WaitFor(“regex:[$%>;] $”).  We then use $cmdresult to capture the result of running an SSH command via our connection to our current ESXi host. To finish capturing the output of the command on our remote host, we use a regex check again (“regex:[~#]”). You’ll notice that the actual command is also on this line, i.e. line 18 (openssl). After the result is captured, we disconnect our SSH session, then proceed to build our small report array. At the end of the script, we output the $Report array to the screen.

 

Results of the above script in PowerCLI

 

As far as I can see, the wodSSH library can open up a whole new world of automation for you in the data centre. Think of the possibilities – you could automate just about everything, from switches, routers, Linux/Unix hosts / ESXi Hosts to iLO / DRAC and other management cards. Essentially, anything that accepts commands via SSH! If you have any automation stories using wodSSH, or just PowerShell or PowerCLI in general, I would love to hear about them – so feel free to drop a comment or two on this post.