Comparing CDN Performance

Share

View Part 2 Here

More and more CDN providers are appearing with competing prices and features. Here I’m going to compare Amazon Cloud Front, Rackspace Cloud Files, and Amazon S3.

First off, Amazon S3 isn’t really a CDN per se, but I’ve ran across quite a few sites comparing Rackspace Cloud to S3, which isn’t a fair comparison. Amazon S3 does, however, have its own strong points which we will get into later.

Response Time

We will start by taking a look at the response time for the three services over a period of 1 month

The average response time for Amazon S3 was 819ms which is pretty bad. However, S3 was never really meant to be a CDN, this is what their Cloud Front service is for. After looking at the Cloud Front average response time we can see it to be 310ms, a much better response time. And then there’s Rackspace Cloud Files, coming in at 210ms, which is even better. Rackspace Cloud Files uses Limelight for their CDN solution which is one of the industry leaders in content delivery, up there with Akamai. I would have done a comparison to Akamai, however I lack the funds to even get close to that. Rackspace Cloud makes it easy to be able to use the “big guys” infrastructure without having to pay the big price.

Pricing

Here’s a quick breakdown of pricing for Amazon and Rackspace CDN services.

Amazon Rackspace
Storage Costs $0.15 / GB† $0.15 / GB
Bandwidth Out $0.17 / GB† (US Traffic‡) $0.22 / GB
Bandwidth In Free until June 30th 2010* $0.08 / GB
PUT/POST/LIST Requests $0.01 per 1,000 $0.01 per 500 Free for files over 250k
GET Requests $0.01 per 10,000 (US Traffic‡) Free
DELETE Requests Free Free

*After June 30th inbound data transfer is $0.10 / GB †Variable Cost; Decreases as more space or bandwidth is used. ‡Higher charges apply for other regions such as Japan. Click here to view the full pricing model.

Keep in mind with Amazon, you’re paying the transfer cost to transfer your file from their S3 storage to various end points around the globe. Rackspace has no such charge. Due to amazons tiered pricing, if you needed to store and deliver hundreds to thousands of terabytes of data a month, then Amazon would be cheaper for you. But then again, if you had that much volume, chances are you don’t care what I think is best for you and you defiantly don’t need help picking the right CDN. You can view the cost calculator for both services here. Rackspace Cloud FilesAmazon Web Services Keep in mind when you setup the Amazon Web Services monthly calculator you will need to add S3 and Cloud Front. The S3 account will hold all of your data, and the outbound bandwidth will vary depending on how many files you have, their sizes, and how often the Cloud Front endpoints refresh their cache of your data. Cloud Front will then be where you allocate your total projected outbound bandwidth and average object size (which is to calculate GET requests I’m guessing) as well as allocate percentages to different regions, since Amazon charges differently per GB depending on the region the server responding to the request is located.

Speed

Next up is a speed test. I’ve uploaded a 100MB file to Amazon S3 and Rackspace Cloud Files to download from various locations. Listed below is the time taken (in seconds) to download the 100MB file.

Network Amazon Rackspace
Roadrunner (Central Florida (Residential)) 52 Seconds (1.92 MB/s) 59 Seconds (1.71 MB/s)
Rackspace (Texas) 2.1 Seconds (47.2 MB/s) 1.9 Seconds (45.7 MB/s)
Amazon (N. Virginia) 7.4 Seconds (13.3 MB/s) 15 Seconds (6.13 MB/s)
The Planet (Texas) 8.9 Seconds (11.2 MB/s) 9.3 Seconds (10.8 MB/s)

It’s clear the Roadrunner residential connection was the bottle neck in both downloads. However, when tested from Rackspace Cloud Servers, both Amazon Cloud Front and Rackspace Cloud Files performed quite well. In the test using Amazon EC2, it’s clear that Amazon was the winner. However, the speeds were MUCH worse than when accessed from Rackspace’s network. This leads me to believe that the only reason Amazon won that test was because of their extremely close proximity to their CDN servers. Both providers were faster when pulling from there own network, this was no surprise. However, when pulled from a separate network, in this case Roadrunner and The Planet, Amazon scored slightly faster.

Features

What’s a CDN without features? Well, it’s just a CDN. But hey, that’s what a CDN does, deliver content. Anything above and beyond that is an added bonus. Lets go over the various features Amazon and Rackspace have to offer.

Feature Amazon Rackspace
Maximum File Size 5GB 5GB
Free Storage->Endpoint transfer No Yes
HTTPS S3 Only, Wildcard Certificate No
Detailed Logs Yes Yes
Custom CNAME Yes In the works
Custom Response Headers Yes No
Torrent Support S3 Only No

Two really nice features of Amazon S3 are HTTPS and Torrents. Now keep in mind these are S3 features only, meaning they don’t apply to Amazon’s CDN service Cloud Front. They use a wildcard certificate on the domain *.s3.amazonaws.com so you can use HTTPS to deliver content over S3, although most browsers will give you an invalid certificate warning (ssl_error_bad_cert_domain on firefox). Their other nice feature is torrent support. If you add “?torrent” to the end of any S3 URL, the server will return a .torrent file instead of the file requested. This is especially great for large files since you will have other users helping you out with distribution, lowering your bandwidth cost. The torrent uses Amazons’ tracker and their S3 machines are seeding the file. This way your torrent will never die, so long as the file remains on the S3 servers. Once again, while this isn’t a feature of their CDN service, it’s still a feature of S3, which is required to use their CDN service.

Conclusion

There are many content delivery services out there. Limelight, Akamai, Amazon, Level 3, and many others. Not all of us can afford the nice ones such as Akamai or Level 3, but that’s why there’s amazing pay-as-you-go solutions such as the two mentioned in this article, Amazon and Rackspace. There are others, I just haven’t had the time to try them all. It’s still a very close call between these two CDN’s. They both have their strong points and weaknesses. However, knowing that Rackspace Cloud Files uses Limelight Networks to deliver content really caught my attention. It really depends on how you’re using the service to determine which one is the most affordable. Amazon’s prices are a tad bit cheaper, starting at $0.17 per GB of outbound transfer inside the United States but you’ll be paying for S3 to Cloud Front transfers, as well as GET requests. Amazon also offers the ability to set custom response headers, allowing you to set custom cache settings for various content. When it comes down to it, most of the extra features that Amazon has over Rackspace don’t really apply to Cloud Front, their CDN service. These include torrents, HTTPS, and URL’s that expire after a certain time. Comparing apples to apples, Amazon has the leg up with custom CNAME support and custom response headers at the cost of having to pay for S3 to Cloud Front transfer. If you need a robust solution and plan on transferring extremely massive amounts of data, then Amazon might be your best bet. But if it’s ease of use and bandwidth provided by Limelight, one of the industry leaders in CDN, then Rackspace Cloud Files may be best for you.

 
  • http://s3.cloudberrylab.com Andy

    I always enjoy learning what other people think about Amazon Web Services and how they use them. Check out my very own tool CloudBerry Explorer that helps to manage S3 and CloudFront on Windows . It is a freeware. http://s3.cloudberrylab.com/

  • http://s3.cloudberrylab.com Andy

    I always enjoy learning what other people think about Amazon Web Services and how they use them. Check out my very own tool CloudBerry Explorer that helps to manage S3 and CloudFront on Windows . It is a freeware. http://s3.cloudberrylab.com/

  • http://topsy.com/tb/bit.ly/7CqP1j Tweets that mention Comparing CDN Performance | Ryan Kearney — Topsy.com

    [...] This post was mentioned on Twitter by Ryan Kearney, Lead Lap M System. Lead Lap M System said: Comparing CDN Performance | Ryan Kearney: Posted in Technology | No comments. I threw together a quick video tu.. http://bit.ly/7X79kf [...]

  • http://ryankearney.com/ Ryan Kearney

    This is by far one of the best management systems for Amazon S3 and Cloud Front allowing you to enable or disable logging, manage CNAME’s, modify response headers, and more. I use this tool quite frequently.

  • http://ryankearney.com/ Ryan Kearney

    This is by far one of the best management systems for Amazon S3 and Cloud Front allowing you to enable or disable logging, manage CNAME’s, modify response headers, and more. I use this tool quite frequently.

  • http://www.ubervu.com/conversations/ryankearney.com/2009/12/comparing-cdn-performance/ uberVU – social comments

    Social comments and analytics for this post…

    This post was mentioned on Twitter by rnawky: Comparing @amazonaws and @rackcloud CDN Performance. http://bit.ly/7CqP1j...

  • http://www.rackaid.com/ jeffatrackaid

    Another provider in the low-cost CDN front is SoftLayer. I’ve used them with some clients. They use Internap’s CDN services. They have a “origin-pull” service that is pay-go. I’ve not explicitly tested performance yet, but could be interesting to do.

    I like origin pull because you simply setup a CNAME and the backend pulls the content from your servers. No special setup.

  • http://www.rackaid.com/ jeffatrackaid

    Another provider in the low-cost CDN front is SoftLayer. I’ve used them with some clients. They use Internap’s CDN services. They have a “origin-pull” service that is pay-go. I’ve not explicitly tested performance yet, but could be interesting to do.

    I like origin pull because you simply setup a CNAME and the backend pulls the content from your servers. No special setup.

  • http://ryankearney.com/ Ryan Kearney

    I took a look at SoftLayer. Looks like they want you to contact sales in order to get pay-as-you-go pricing so I’ll have to look at that later. Pricing looks a bit hefty though. $2/GB of storage and $0.20/GB of transfer out.

  • http://ryankearney.com/ Ryan Kearney

    I took a look at SoftLayer. Looks like they want you to contact sales in order to get pay-as-you-go pricing so I’ll have to look at that later. Pricing looks a bit hefty though. $2/GB of storage and $0.20/GB of transfer out.

  • http://www.scoremysore.com/ Fmaous Auns

    The MB/s for RoadRunner x Amazon in the Speed section should be 1.58 MB/s, assuing the 63 download time is correct.

  • http://www.scoremysore.com/ Fmaous Auns

    The MB/s for RoadRunner x Amazon in the Speed section should be 1.58 MB/s, assuing the 63 download time is correct.

  • http://rupertrawnsley.blogspot.com Rupert Rawnsley

    Great article. There is a real absence of any intel about these issues, especially bitrates.

    We recently identified a problem with our international bandwidth and are looking at CDN solutions. I will blog the results eventually, but I am currently testing Azure CDN against CloudFront (and a standard UK-based host). We are mounting a 1MB file on each service then using website monitoring to estimate bitrate. The only monitoring system I have found so far that offers this possibilty is http://www.websitepulse.com who report the full download time, not just the inital response time. They also have a big range of international monitoring stations as well. The pricing is off the hook, but they have a 14 day free trial.

    If you still have your RackSpace account, could you host our file so I can set up another monitor? You can find it here: http://OvertureOnline.com/Data/SpeedTest/Test1MB.zip. It isn’t a valid ZIP file, it just has that extension to suppress HTTP compression. Drop me an email if you do and I will keep you posted with the comparison.

  • http://rupertrawnsley.blogspot.com Rupert Rawnsley

    Great article. There is a real absence of any intel about these issues, especially bitrates.

    We recently identified a problem with our international bandwidth and are looking at CDN solutions. I will blog the results eventually, but I am currently testing Azure CDN against CloudFront (and a standard UK-based host). We are mounting a 1MB file on each service then using website monitoring to estimate bitrate. The only monitoring system I have found so far that offers this possibilty is http://www.websitepulse.com who report the full download time, not just the inital response time. They also have a big range of international monitoring stations as well. The pricing is off the hook, but they have a 14 day free trial.

    If you still have your RackSpace account, could you host our file so I can set up another monitor? You can find it here: http://OvertureOnline.com/Data/SpeedTest/Test1MB.zip. It isn’t a valid ZIP file, it just has that extension to suppress HTTP compression. Drop me an email if you do and I will keep you posted with the comparison.

  • http://ryankearney.com/ Ryan Kearney

    Good catch, I’m going to re-run the test to figure out which one was correct.

  • http://ryankearney.com/ Ryan Kearney

    Good catch, I’m going to re-run the test to figure out which one was correct.

  • mike

    Hey any chance of you doing other providers like SimpleCDN or GoGrid CDN?

  • mike

    Hey any chance of you doing other providers like SimpleCDN or GoGrid CDN?

  • http://ryankearney.com/ Ryan Kearney

    I probably will, just signed up for SimpleCDN actually and it looks really good so far. Haven’t checked out GoGrid yet though. Maybe some time next weekend.

  • http://ryankearney.com/ Ryan Kearney

    I probably will, just signed up for SimpleCDN actually and it looks really good so far. Haven’t checked out GoGrid yet though. Maybe some time next weekend.

  • Charles

    Good article Ryan however I have one point of clarification I hope you could address – I understand testing the services from a ISP such as Roadrunner and testing the download speed from Amazon or Rackspace…however when you test Rackspace – Rackspace or Amazona- Amazon….I am not seeing how you are doing this. Most people are not going to be inside either “network” as they are not ISP’s so perhaps I am missing something completely?

  • Charles

    Good article Ryan however I have one point of clarification I hope you could address – I understand testing the services from a ISP such as Roadrunner and testing the download speed from Amazon or Rackspace…however when you test Rackspace – Rackspace or Amazona- Amazon….I am not seeing how you are doing this. Most people are not going to be inside either “network” as they are not ISP’s so perhaps I am missing something completely?

  • http://ryankearney.com/ Ryan Kearney

    I used Amazon EC2 services and Rackspace Cloud Servers. Both companies offer VPS services as well as CDN services. I made started up a pay by the hour server on both networks to test download speeds.

  • http://ryankearney.com/ Ryan Kearney

    I used Amazon EC2 services and Rackspace Cloud Servers. Both companies offer VPS services as well as CDN services. I made started up a pay by the hour server on both networks to test download speeds.

  • http://www.m80im.com JasonM80

    Great job on the comparison!

    I am curious to hear about Rupert’s comparison between Azure CDN, CloudFront, and the UK-based host. I found this article which does a comparison between Azure CDN and CloudFront, though it does not address performance: http://bit.ly/8cUeFA.

    As for a web-monitoring tool, one solution you might consider is Microsoft’s System Center Operations Manager, which has web-monitoring capability: http://bit.ly/70JSp5. Since it is intended to do much more than web-monitoring, it might be overkill for what you are trying to do.

    (I am contracted by M80, working with Microsoft to promote Windows Azure)

  • http://www.m80im.com JasonM80

    Great job on the comparison!

    I am curious to hear about Rupert’s comparison between Azure CDN, CloudFront, and the UK-based host. I found this article which does a comparison between Azure CDN and CloudFront, though it does not address performance: http://bit.ly/8cUeFA.

    As for a web-monitoring tool, one solution you might consider is Microsoft’s System Center Operations Manager, which has web-monitoring capability: http://bit.ly/70JSp5. Since it is intended to do much more than web-monitoring, it might be overkill for what you are trying to do.

    (I am contracted by M80, working with Microsoft to promote Windows Azure)

  • Sam

    Ryan, I’m looking for CDN services, and I found this comparison useful. I’m looking forward to your expanded comparison of additional CDN services, is there a date you expect to publish it?

  • Sam

    Ryan, I’m looking for CDN services, and I found this comparison useful. I’m looking forward to your expanded comparison of additional CDN services, is there a date you expect to publish it?

  • http://ryankearney.com/ Ryan Kearney

    Tomorrow, December 15th some time around 8PM.

  • http://ryankearney.com/ Ryan Kearney

    Tomorrow, December 15th some time around 8PM.

  • Bozo

    How do you define “response time”? Time to first byte? Time to HEAD a file? Something else?

  • Bozo

    How do you define “response time”? Time to first byte? Time to HEAD a file? Something else?

  • http://www.ryankearney.com Ryan Kearney

    I used Pingdom to monitor the response time. As to what they actually monitor, I’m not entirely sure.

  • http://www.ryankearney.com Ryan Kearney

    I used Pingdom to monitor the response time. As to what they actually monitor, I’m not entirely sure.

blog comments powered by Disqus