how to Archives - Page 5 of 10

Simple Content Delivery Network (CDN) using Amazon AWS (S3 + CloudFront)

June 16, 2013 by Sean

Content Delivery Networks

Having a content delivery network has many benefits for your users or clients. One of the most obvious reasons of having a CDN, is the ability to serve up content to your users from multiple (often the most optimal) locations. Users access files that originate from one original source location, but the content is delivered by the closest location(s), often with the lowest latency and highest possible speed.

Using Amazon CloudFront, you can share dynamic, static, or even streamed content to users (including full websites), using Amazon’s global network of edge locations. This means that content can be served to users at the highest possible speeds, with the lowest possible latencies. In this blog post, I will cover the steps you need to take to deploy a basic CDN using Amazon AWS. For this purpose, we will leverage a combination of Amazon S3 + CloudFront.

Setting up Amazon S3

Amazon S3 (Amazon Simple Storage Service) is essentially Amazon’s “storage for the Internet”, and as explained above, CloudFront is a content delivery network service. As such, both products sit in Amazon’s “Storage & Content Delivery” stack.

To get started you will of course need an Amazon AWS account. Go to http://aws.amazon.com/ and register. You will need to provide credit card details, but most products have some sort of free tier that you can utilise for initial testing (usually free for up to 1 year, based on certain utilisation thresholds).
Once you are all signed up, you’ll need to navigate to the AWS Web Console. This is the central location you can use to manage all AWS services (among other options such as the AWS SDK and Command Line).

aws-console-example — The central, AWS Web Management Console

To start, we’ll need to define an origin location for our content. This is the location our original files are kept. For this purpose, we will use Amazon S3. It allows us easy access to files that we place in something Amazon call a “bucket”. I like to think of it as a folder, or container. You can have as many buckets as you wish, however each one’s name needs to be completely unique across Amazon S3. Click on “S3” under the “Storage & Content Delivery” heading of your AWS Console to get started.
From here, you will be greeted with a welcome page and some explanation of what S3 is. Simply click “Create Bucket” to get going.

Provide a unique bucket name, and specify a region to use. Regions have the benefit of allowing organisations to comply with storage regulation rules – for example, if you were storing client data that you were bound legally to keep within the UK, you would specify the Ireland region.

Your new bucket will appear in the S3 Management Console after being created. Simply click the name of the bucket to open it. For our simple CDN, we’ll just be serving up one single file – pretend this was a really large file that needed efficient distribution to many people – for example a large media file. At the top left, you’ll see an “Upload” button. Click this, and choose a file to upload as your test file. I will be using a simple image file. (By the way, Amazon have a service called “Amazon Import/Export”, which allows you to send really large amounts of data via post on portable media to Amazon for them to upload directly to your Amazon S3 or Glacier services).
Click “Start Upload” once you have chosen a file to test with.
After the file is finished uploading, it will appear in the console under your bucket name. (I called mine “image-for-distribution.png”).

Right-click the file, and choose the option “Make Public” for this test. This choice would be affected by the nature of the files you would want to deliver to users in your own configuration, but for this simple example, this is what I am choosing.
Right-click the file again, and choose “Properties“. Here you can get the direct, public link to your file and test access to it in your web browser. This is simple, direct access, and is not the access we are aiming for, as we will utilise our CDN with CloudFront to serve the file in our final configuration. This is just to test that the direct link is working.

Setting up CloudFront and your Distribution

Now that we know our basic file is being correctly served from Amazon S3, we’ll navigate to “CloudFront” from the main AWS Console (aws.amazon.com). A quick way to get there is by clicking the orange cube icon in the top left of your AWS page – wherever you are in the console, it’ll take you back to the main AWS console. From there just click “CloudFront“.
In CloudFront, we’ll want to create something called a “Distribution“. Click the “Create Distribution” button to get started.

Make sure you select “Download” type for the “delivery method” when asked on the next page, then click “Continue“.

We’ll now select various options for our CloudFront Distribution.
- For “Origin Domain Name“, click the text box and you’ll see a populated list of Amazon S3 buckets. Your bucket you created earlier should feature here. Click it to select it.
- The “Origin ID” should auto populate based on your S3 bucket name you chose.
- If you wish to restrict users to only access your content via CloudFront URLs, and not direct by S3 URLs, then choose “Yes” for “Restrict Bucket Access“.
- If you chose “Yes” for restricting bucket access, you’ll also need to create a “Comment” and “Grant Read Permissions” on the bucket for CloudFront’s access to the S3 bucket. Click “Yes, Update Bucket Policy” to have CloudFront get read access automatically to the S3 bucket.
- Select “HTTP and HTTPS” for “Viewer Protocol Policy“.
- You can customise the object caching properties if you wish, but for this example, just leave the “Default Cache Behavior Settings” on their defaults.
- Now you can set your “Distribution Settings“. Choose “Use All Edge Locations (Best Performance)” for “Price Class“. This will ensure that all edge locations around the world are used to distribute your content in the fastest, most efficient way to your users. You could also restrict this to other groups of regions e.g. only the US and Europe for example – this would be a cheaper option, but not as efficient for all users globally.
- Next, we can add an alternate CNAME for the distribution. This is highly recommended so that you can provide your own domain name formatted URLs to users, instead of a long, ugly default Amazon CloudFront URL. Enter something now, (for example I will use cdn.shogan.co.uk as I own the domain and can create this CNAME record myself in DNS). Once you are complete with this distribution setup, you should get the Distribution URL, and point a new CNAME record to the full URL that CloudFront assigns to your distribution.
- Leave all other options at their defaults for now, and make sure that the last option “Distribution State” is “Enabled“, then click the “Create Distribution” button at the very bottom.

Your Distribution should now be created. Use the Navigation menu on the left side of the screen and click “Distribution” to see a list of your CloudFront Distributions.

At first the “Status” will show “InProgress“. After a few minutes this should change to “Deployed“.
In the mean time, look for your “Domain Name” that this Distribution has been assigned, and go and create a CNAME record pointing the CNAME you specified when creating this distribution, to the domain name. For example, you may have something like dxxxxxxxxxm.cloudfront.net. In my case, I specified a CNAME of cdn.shogan.co.uk, so I will create a CNAME record linking these together.

Testing

Once your CNAME record is created, type in your new CNAME record, followed by a forward slash, and then the name of the file you originally uploaded to your S3 bucket that is linked to by this CloudFront distribution. For example, my file was called “file-for-distribution.png” and my CNAME record I made is cdn.shogan.co.uk. So to utilise my CloudFront CDN, I would simply access the file as “cdn.shogan.co.uk/image-for-distribution.png”. If your DNS takes a while to apply/propagate, then you can simply use the CloudFront domain name assigned to your Distribution (for example dxxxxxxxxxxm.cloudfront.net/yourfilename.extension) to test out your distribution. Remember to ensure your distribution is in a deployed state before testing. You should now see your file served up in your web browser via your brand spanking new Amazon AWS powered CDN!

Conclusion

That concludes the basic setup of a Amazon S3 + CloudFront powered Content Delivery Network. I hope this was useful for some. In forthcoming blog posts I will delve into setting up custom logging and monitoring / alerting for your CDN. Please remember to like/share/tweet this post out to friends if you thought it was useful.

PowerCLI – creating port groups and specifications to modify port groups

September 14, 2012 by Sean

I wanted to quickly create some standard VM port groups across a particular vSwitch for all hosts in my lab / testing environment at work. Since I was using Standard vSwitches and not a dvSwitch, I didn’t feel like using the GUI to create these on every individual ESXi host. In addition to creating the port group on each vSwitch, I also wanted to change the security policy on each for Promiscuous mode to “Accept”. The reason for this being that this port group is going to be used to run virtual nested ESXi hosts, and this is required to allow nested VMs to communicate on the network.

So the obvious solution here for me was to create a quick PowerCLI script to create these port groups on all hosts and set the security option for each too. Here is the script:

$vSwitch = "vSwitch0"
$portgrpname = "vInception Portgroup"
$AllConnectedHosts = Get-VMHost | Where {$_.ConnectionState -eq "Connected"}

foreach ($esxihost in $AllConnectedHosts) {
	$currentvSwitch = $esxihost | Get-VirtualSwitch | Where {$_.Name -eq $vSwitch}
	New-VirtualPortGroup -Name $portgrpname -VirtualSwitch $currentvSwitch -Confirm:$false
	$currentesxihost = Get-VMHost $esxihost | Get-View
	$netsys = Get-View $currentesxihost.configmanager.networksystem
	$portgroupspec = New-Object VMWare.Vim.HostPortGroupSpec
	$portgroupspec.vswitchname = $vSwitch
	$portgroupspec.Name = $portgrpname
	$portgroupspec.policy = New-object vmware.vim.HostNetworkPolicy
	$portgroupspec.policy.Security = New-object vmware.vim.HostNetworkSecurityPolicy
	$portgroupspec.policy.Security.AllowPromiscuous = $true
	$netsys.UpdatePortGroup($portgrpname,$PortGroupSpec)
	
}

Keep in mind that this script will create the port group on “vSwitch0” – change this if your vSwitch that is hosting this port group on each host is named differently. It will obviously rely on this vSwitch existing to work. You can also modify the $portgrpname to your own choice of course.

Lastly, you can easily modify this script to change other Security options for the new port group, as the port group specification has already been created in this script. Just use the $portgroupspec.policy.Security object to add other specifications.

Enjoy!

Changing Registry entries on multiple systems with PowerShell and Remoting

June 12, 2012 by Sean

A few weeks ago, a colleague asked if I knew of a way to script the change or modification of the Registered Owner / Organization information on a Windows Server system (2003 or 2008). I knew that this could be achieved with PowerShell and had some initial ideas, so I spent a few minutes whipping up the script below.

For this to work, you should ideally have all systems on the same Windows Domain and have enabled PowerShell remoting on each system that needs to be changed. Of course you could also just run the script on a single workstation/server on its own without the need for PSRemoting.

# On all remote machines that need their info changed
Set-ExecutionPolicy RemoteSigned
Enable-PSRemoting # Say yes to all prompts

#region This part only needed if machines do not belong to the same domain...
# Note: This can be a security risk, only use if you are sure you want to allow any host as a trusted host. (e.g. fine for lab environments)
cd wsman::localhost\client
Set-Item .\TrustedHosts * # Say yes to all prompts
#endregion

# Run on your management machine/machine you are using to update all others...
$computers = @("SERVER1","SERVER2","SERVER3")

foreach ($computer in $computers) {
    Enter-PSSession $computer
    cd 'HKLM:\Software\Microsoft\Windows NT\CurrentVersion'
    Set-ItemProperty -Path . -Name "RegisteredOwner" -Value "Auth User"
    Set-ItemProperty -Path . -Name "RegisteredOrganization" -Value "Lab"
    Exit-PSSession
}

So the above should update your registered owner and organization details for each server listed in the $computers array. (Specify your own host names here). The above script should be easy enough to modify if you are looking to change other registry entries. Finally, don’t forget that you should always be careful when updating registry, especially via script – make sure you have backups!

Finding Vendor specific VIBs on ESXi hosts with PowerCLI

March 30, 2012 by Sean

get-esxcli-vib-example — Example showing VIBs loaded on a host with a search of the vendor name "VMware"

The other day I was trying to find a list of Custom VIBs (VMware Installation Bundles) that were installed on an ESXi host. The reason was that I just wanted to verify that the VIB had actually installed correctly or not. I threw the query out on Twitter and of course @alanrenouf had a solution in next to no time.

So Alan’s solution is to use the Get-EsxCli cmdlet and specify the host name using the VMHost parameter. After that, he simply uses the code property “software” to gain access to the list of VIBs on the host. E.g.

$ESXCLI = Get-EsxCli -VMHost esxi-01.noobs.local
$ESXCLI.software.vib.list()

I have used esxcli on its own before but didn’t realise that PowerCLI had this cmdlet built in to interface with hosts in the same way that esxcli would. This is a great solution and means you can fetch so much more in this regard.

To filter things down a bit more and find the exact match for the Dell OMSA VIB I was looking out for, I used a where clause looking for a match for “dell” on the Vendor property:

$ESXCLI.software.vib.list() | Select AcceptanceLevel,ID,InstallDate,Name,ReleaseDate,Status,Vendor,Version | Where {$_.Vendor -match "dell"}

Thanks again to Alan Renouf for pointing out the use of Get-EsxCli and for providing an example!

How to use PoSH or PowerCLI to SSH into network devices – wodSSH [Part 2]

March 26, 2012 by Sean

In [part 1] of this series of posts, I explored SharpSSH, and how to use this to SSH into network devices via PowerShell or PowerCLI. I found this extremely useful, and managed to automate a process of retrieving some SHA1 SSL fingerprints from a few clusters of ESX hosts in the process. I did however find that loading the dll and functions necessary to get this bit of automation done was a little tedious. I had heard about wodSSH via the get-scripting podcast and decided I would need to take a look at this.

If you are planning on using wodSSH, you should start by downloading it here. Note that you should definitely buy a license if you are going to be using this for your job or day to day automation – the license is not to expensive if you consider the kinds of automation you could achieve with this! For this blog post I am just using the trial version in my lab environment at home to try out – the trial pops up a “nag screen” every now and then to indicate you are unlicensed. Once downloaded, install wodSSH and the libraries will now be available for use on your machine.

Download the script below, or take a look at it in the Syntax Highlighted area below – this is an example of how I was able to SSH into a cluster of ESXi hosts and run a remote command via SSH through PowerCLI. In this example, a table is returned with Host names, along with their SHA1 fingerprint information. It is possible to retrieve this info (as of ESXi 5 I believe) in PowerCLI, but for this example I just wanted to demonstrate the use of SSH and PowerShell / PowerCLI.

[download id=”14″]

$login = "root"
$password = "yourrootpassword" 

$Report = @()
$VMHosts = Get-VMHost | Where {$_.ConnectionState -eq "Connected"} | Sort Name

foreach ($vmhost in $VMHosts) {
	$hostname = $vmhost
	$ssh = New-Object -ComObject "weOnlyDo.wodSSHCom.1"
	$ssh.Hostname = $hostname
	$ssh.Login = $login
	$ssh.Password = $password
	$ssh.Blocking = 1
	$ssh.Protocol = 4
	$ssh.Timeout = 25
	$ssh.Connect($hostname)
	$ssh.WaitFor("regex:[$%#>] $")
	$cmdresult = $ssh.Execute("openssl x509 -sha1 -in /etc/vmware/ssl/rui.crt -noout -fingerprint`r`n","regex:[~#]")
	$ssh.Disconnect()
	$temp = $cmdresult | select-string -pattern "SHA1 Fingerprint="
	Write-Host $cmdresult -ForegroundColor Green
	$row = New-Object -TypeName PSObject -Property @{
		SHA1 = $cmdresult
		HostName = $vmhost
	}
	$Report += $row
}
$Report

You will notice that on line 09, we load the wodSSH library using the New-Object cmdlet. After this we continue to setup some properties for our $ssh connection object. A regex check is used to determine when are connected to a host and have an SSH shell prompt ($ssh.WaitFor(“regex:[$%>;] $”). We then use $cmdresult to capture the result of running an SSH command via our connection to our current ESXi host. To finish capturing the output of the command on our remote host, we use a regex check again (“regex:[~#]”). You’ll notice that the actual command is also on this line, i.e. line 18 (openssl). After the result is captured, we disconnect our SSH session, then proceed to build our small report array. At the end of the script, we output the $Report array to the screen.

wodssh-example — Results of the above script in PowerCLI

As far as I can see, the wodSSH library can open up a whole new world of automation for you in the data centre. Think of the possibilities – you could automate just about everything, from switches, routers, Linux/Unix hosts / ESXi Hosts to iLO / DRAC and other management cards. Essentially, anything that accepts commands via SSH! If you have any automation stories using wodSSH, or just PowerShell or PowerCLI in general, I would love to hear about them – so feel free to drop a comment or two on this post.