Cloud Archives - Page 2 of 2

Simple Content Delivery Network (CDN) using Amazon AWS (S3 + CloudFront)

June 16, 2013 by Sean

Content Delivery Networks

Having a content delivery network has many benefits for your users or clients. One of the most obvious reasons of having a CDN, is the ability to serve up content to your users from multiple (often the most optimal) locations. Users access files that originate from one original source location, but the content is delivered by the closest location(s), often with the lowest latency and highest possible speed.

Using Amazon CloudFront, you can share dynamic, static, or even streamed content to users (including full websites), using Amazon’s global network of edge locations. This means that content can be served to users at the highest possible speeds, with the lowest possible latencies. In this blog post, I will cover the steps you need to take to deploy a basic CDN using Amazon AWS. For this purpose, we will leverage a combination of Amazon S3 + CloudFront.

Setting up Amazon S3

Amazon S3 (Amazon Simple Storage Service) is essentially Amazon’s “storage for the Internet”, and as explained above, CloudFront is a content delivery network service. As such, both products sit in Amazon’s “Storage & Content Delivery” stack.

To get started you will of course need an Amazon AWS account. Go to http://aws.amazon.com/ and register. You will need to provide credit card details, but most products have some sort of free tier that you can utilise for initial testing (usually free for up to 1 year, based on certain utilisation thresholds).
Once you are all signed up, you’ll need to navigate to the AWS Web Console. This is the central location you can use to manage all AWS services (among other options such as the AWS SDK and Command Line).

aws-console-example — The central, AWS Web Management Console

To start, we’ll need to define an origin location for our content. This is the location our original files are kept. For this purpose, we will use Amazon S3. It allows us easy access to files that we place in something Amazon call a “bucket”. I like to think of it as a folder, or container. You can have as many buckets as you wish, however each one’s name needs to be completely unique across Amazon S3. Click on “S3” under the “Storage & Content Delivery” heading of your AWS Console to get started.
From here, you will be greeted with a welcome page and some explanation of what S3 is. Simply click “Create Bucket” to get going.

Provide a unique bucket name, and specify a region to use. Regions have the benefit of allowing organisations to comply with storage regulation rules – for example, if you were storing client data that you were bound legally to keep within the UK, you would specify the Ireland region.

Your new bucket will appear in the S3 Management Console after being created. Simply click the name of the bucket to open it. For our simple CDN, we’ll just be serving up one single file – pretend this was a really large file that needed efficient distribution to many people – for example a large media file. At the top left, you’ll see an “Upload” button. Click this, and choose a file to upload as your test file. I will be using a simple image file. (By the way, Amazon have a service called “Amazon Import/Export”, which allows you to send really large amounts of data via post on portable media to Amazon for them to upload directly to your Amazon S3 or Glacier services).
Click “Start Upload” once you have chosen a file to test with.
After the file is finished uploading, it will appear in the console under your bucket name. (I called mine “image-for-distribution.png”).

Right-click the file, and choose the option “Make Public” for this test. This choice would be affected by the nature of the files you would want to deliver to users in your own configuration, but for this simple example, this is what I am choosing.
Right-click the file again, and choose “Properties“. Here you can get the direct, public link to your file and test access to it in your web browser. This is simple, direct access, and is not the access we are aiming for, as we will utilise our CDN with CloudFront to serve the file in our final configuration. This is just to test that the direct link is working.

Setting up CloudFront and your Distribution

Now that we know our basic file is being correctly served from Amazon S3, we’ll navigate to “CloudFront” from the main AWS Console (aws.amazon.com). A quick way to get there is by clicking the orange cube icon in the top left of your AWS page – wherever you are in the console, it’ll take you back to the main AWS console. From there just click “CloudFront“.
In CloudFront, we’ll want to create something called a “Distribution“. Click the “Create Distribution” button to get started.

Make sure you select “Download” type for the “delivery method” when asked on the next page, then click “Continue“.

We’ll now select various options for our CloudFront Distribution.
- For “Origin Domain Name“, click the text box and you’ll see a populated list of Amazon S3 buckets. Your bucket you created earlier should feature here. Click it to select it.
- The “Origin ID” should auto populate based on your S3 bucket name you chose.
- If you wish to restrict users to only access your content via CloudFront URLs, and not direct by S3 URLs, then choose “Yes” for “Restrict Bucket Access“.
- If you chose “Yes” for restricting bucket access, you’ll also need to create a “Comment” and “Grant Read Permissions” on the bucket for CloudFront’s access to the S3 bucket. Click “Yes, Update Bucket Policy” to have CloudFront get read access automatically to the S3 bucket.
- Select “HTTP and HTTPS” for “Viewer Protocol Policy“.
- You can customise the object caching properties if you wish, but for this example, just leave the “Default Cache Behavior Settings” on their defaults.
- Now you can set your “Distribution Settings“. Choose “Use All Edge Locations (Best Performance)” for “Price Class“. This will ensure that all edge locations around the world are used to distribute your content in the fastest, most efficient way to your users. You could also restrict this to other groups of regions e.g. only the US and Europe for example – this would be a cheaper option, but not as efficient for all users globally.
- Next, we can add an alternate CNAME for the distribution. This is highly recommended so that you can provide your own domain name formatted URLs to users, instead of a long, ugly default Amazon CloudFront URL. Enter something now, (for example I will use cdn.shogan.co.uk as I own the domain and can create this CNAME record myself in DNS). Once you are complete with this distribution setup, you should get the Distribution URL, and point a new CNAME record to the full URL that CloudFront assigns to your distribution.
- Leave all other options at their defaults for now, and make sure that the last option “Distribution State” is “Enabled“, then click the “Create Distribution” button at the very bottom.

Your Distribution should now be created. Use the Navigation menu on the left side of the screen and click “Distribution” to see a list of your CloudFront Distributions.

At first the “Status” will show “InProgress“. After a few minutes this should change to “Deployed“.
In the mean time, look for your “Domain Name” that this Distribution has been assigned, and go and create a CNAME record pointing the CNAME you specified when creating this distribution, to the domain name. For example, you may have something like dxxxxxxxxxm.cloudfront.net. In my case, I specified a CNAME of cdn.shogan.co.uk, so I will create a CNAME record linking these together.

Testing

Once your CNAME record is created, type in your new CNAME record, followed by a forward slash, and then the name of the file you originally uploaded to your S3 bucket that is linked to by this CloudFront distribution. For example, my file was called “file-for-distribution.png” and my CNAME record I made is cdn.shogan.co.uk. So to utilise my CloudFront CDN, I would simply access the file as “cdn.shogan.co.uk/image-for-distribution.png”. If your DNS takes a while to apply/propagate, then you can simply use the CloudFront domain name assigned to your Distribution (for example dxxxxxxxxxxm.cloudfront.net/yourfilename.extension) to test out your distribution. Remember to ensure your distribution is in a deployed state before testing. You should now see your file served up in your web browser via your brand spanking new Amazon AWS powered CDN!

Conclusion

That concludes the basic setup of a Amazon S3 + CloudFront powered Content Delivery Network. I hope this was useful for some. In forthcoming blog posts I will delve into setting up custom logging and monitoring / alerting for your CDN. Please remember to like/share/tweet this post out to friends if you thought it was useful.

Troubleshooting the Autolab vCloud Director 1.5.1 installation

February 10, 2013 by Sean

I have had this issue twice now, where deploying vCD via the Autolab PXE boot option on the vCD VM fails. As far as I can tell, the process seems to fail on the Oracle Express DB installation, due to the RPM not being a valid package. The vCloud Director steps seem to be the same for Autolab 1.0 or 1.1, so the following applies to both.

error: /root/oracle-xe-11.2.0-1.0.x86_64.rpm: not an rpm package (or package manifest)

You can see the error I was getting in the screenshot I captured during boot time below. I had checked the RPM file and everything else to ensure it was in place, and indeed it was. Even vCD installs via the script, although it of course does not work due to the database not being there.

Here is the process I used to correct my vCD install.

Allow VM to finish booting, even with the missing oracle DB.
Use PuTTy to SSH to the vCD VM (either direct from your VC or DC VM, or if you have the route setup, from your host machine (in the case you are using VMware Workstation for example). Default credentials are in the Autolab setup guide document
Open up the “Build” share on the NAS VM, and location the vcd-install script. Default location: \\192.168.199.7\Build\Automate\CentOS\vcd-install (open this with a text editor)
Locate the method for each section of the install script. There is a section for each process in the script. For each method, copy out the entire block, paste it into a new text document, and remove any exclamation marks from any “echo” parts of the script. I found that manually tracking through this script using PuTTy gave me issues with the exclamation marks being misinterpreted by the shell, so I removed these. You’ll need to get a script block for the following sections and do this:
- verify() {}
- installOracle() {}
- configureOracle() {}
- generateCertificates() {}
- installvCD() {}
- configurevCD() {}
Remember to copy the whole block, including the start and end braces {} – paste these into a new text document, remove the exclamation marks, then copy-paste them back into your shell open in PuTTy. Hit enter, and the method will be entered and ready for use.
Once all the methods have been copied in, you can simply type the name of the method, followed by enter to execute them. By doing it this way, you can manually step through the process and figure out where any potential remaining issues may be. This script is normally executed during the PXE boot installation process so you don’t really get a chance to slowly track through it.
Type each method in until you reach and complete the last “configurevCD” one
- verify
- installOracle
- configureOracle
- generateCertificates
- installvCD
- configurevCD
You may find that the generateCertificates and installvCD methods complete and echo out that they had already been completed prior – this is fine.
After configurevCD finished, all being well, you should now have vCD started, and you should be able to browse over to https://vcd.lab.local and finish the initial configuration via the vCD web page.

Other tips to try would be to:

MD5 hash check the RPM of the Oracle Express database that you download and place in your Build share – make sure it is not a corrupted file
Ensure you have the correct version of vCD and the Oracle Express database downloaded

The latest trends in VMware and Cloud Computing

August 29, 2012 by Sean

VMware promotes virtualization as a catalyst for cloud computing. Cloud infrastructures are built on and powered by VMware. VMware allows IT professionals to build solutions that are specifically tailored to a client’s individual needs. Internal and external clouds may be created to handle the needs of a growing business. Hybrid clouds are growing in popularity for businesses that want the convenience of both. Here are some of the benefits of VMware cloud virtualization:

Efficient Processes. VMware makes it possible to automate processes and employ utilization to increase IT performance. When IT professionals leverage existing resources and avoid expenses related to infrastructure investment, the total cost of ownership (TCO) is reduced tremendously.
Agility. End-users gain a more secure environment with cloud computing. With VMware, IT professionals can be assured that they will preserve IT authority, control and security while remaining compliant. Processes are also simplified to make the job easier. An IT organization is able to respond quickly to organizations with evolving business needs.
More Flexibility. IT professionals can use VMware in conjunction with traditional systems for maximum flexibility. The systems may be deployed internally or externally. When configuring VMware, IT professionals are not limited to using any one vendor or technology. The solutions are portable and are capable of using a common management and security framework.
Better Security. VMware solutions protect end-points, the network edge and applications through virtualization. The cloud based deployments of security patches and solutions are dynamic and constantly being updated.
Automation and Management. With VMware, a highly efficient, self-managing infrastructure can be created. Business rules and policies can be mapped to IT resources when the tools are virtually pooled.
Portable and Independent. Open standard VMware solutions provide more flexibility and reduce the dependence on a particular vendor. With this security model, applications are easily portable from internal datacenters to external service provider clouds. The applications are also dynamic, optimized and deployable on public clouds with VMware cloud application platforms.
Saves Time. A self-service cloud-based portal is capable of reducing time spent by deploying standardized solutions that have been pre-configured to operate off-the-shelf or out-of-the-box. This method promotes efficiency through automation and standardization. Tailored services are also popular and can be achieved with VMware solutions. IT can remain in compliance and preserve control over policies with VMware.
Virtual Pooling and Dynamic Resource Allocation. Virtual datacenters are created by pooling IT resources through abstraction. Logical storage building blocks, server units and network are integrated into the solution to power applications. This process is completed in accordance to regulations and business rules. User demand also plays a role in how these applications are deployed and hosted.

How Businesses are using VMware to transition to the Cloud

Dynamic businesses have a need for a robust and affordable IT solution. Most businesses use 70 percent of their resources focusing on maintenance of servers and applications in a traditional system. With only 30 percent of the IT budget left for innovation, companies cannot grow and provide the type of service and products its clients need and desire. IT management is searching for a better strategy, and VMware seems to be a viable solution.

VMware provides users with faster response times. Faster response times lead to lower costs over time. Self-managed virtual infrastructures are efficient and preferred by many businesses.

IT professionals can identify which cloud-based solution is best for your company. The choices typically consist of a public, private or hybrid solution. Many companies have successfully implemented these solutions.

VMware’s cloud infrastructure and management application is commonly known as vCloud Director. This application will allow a company to transition to the cloud at their own pace. The application was introduced in 2011 to provide companies with greater flexibility and efficiency in the cloud.

VMware’s solution allows companies the ability to leverage their existing infrastructure. This saved business owners significant time and money. The savings could then be reinvested for innovation. VMware’s cost-effective solution provides an answer to the pre-existing solution of 70 percent spending on infrastructure maintenance.

NetApp has exceptional backup and recovery capabilities that are necessary for any company’s disaster recovery solution. Within minutes, VMware’s vCloud Director can recover data. The backup and recovery system is customizable, fast and accurate.

NetApp and VMware have a 24 hour per day and seven day per week global staff monitoring the applications and data stored in the cloud. This ensures the data is protected. Technical support constantly works with all parties to ensure issues are addressed promptly and efficiently. Additionally, VMware ensures that resources are available to meet service level agreements.

Consider How VMware Can Help Your Organization

VMware is a viable solution that can be beneficial in any organization. Consider VMware for your business and witness an increase in productivity, efficiency and mobility. VMware solutions are chosen frequently because they work.

Author Bio:

David Malmborg works with Dell. When David is not working, he enjoys spending time with his two kids. For more information on cloud computing, David recommends clicking here.