Friday, July 6, 2012

Measuring CDN Performance With WebPagetest

When I was at Velocity I heard about a quick and useful trick you can do with WebPagetest to measure the effectiveness of your CDN.  The steps are pretty simple:
  1. Test a URL on your site from a location that is far from your origin.
  2. Using the scripting tab in WebPagetest, point your CDN domains at your origin IP, and run another test.
  3. Compare the results, and see how much your CDN is helping you!
Let's break this down for a Wayfair URL.

Step 1:  Test a URL on Your Site Normally

Since we only have our static content on our CDN, I chose a URL that has a lot of images on it, what we call a "superbrowse" page - http://www.wayfair.com/Outdoor-Wall-Lights-C416512.html.  Since our origin is in the Boston area, I picked the LA node in WebPagetest to give our CDN the best chance of success.  To try and smooth out the results, I configured the test to run 10 times.  It also helps to login and label your test so you can easily find it later.  


While this was running I moved on to step 2...

Step 2:  Use the WebPagetest Scripting Engine to Point to Your Origin

I set this test up almost the exact same way, except in the script tab I entered the following:

 setDns common.csnimages.com 209.202.142.30  
 setDns common1.csnimages.com 209.202.142.30  
 setDns common2.csnimages.com 209.202.142.30  
 navigate http://www.wayfair.com/Outdoor-Wall-Lights-C416512.html  

This points all of our image domains (we are using domain sharding for performance) at our origin IP address and bypasses our CDN entirely.

At this point I just had to wait for the tests to complete and compare the results.

Step 3:  Comparing and Interpreting Results

Here are the test results over 10 runs:

Test Type Mean Median Standard Deviation
With CDN 6.7055 5.744 1.74693
Without CDN 6.758 6.221 1.61

Based on these results, it appears that our CDN isn't actually providing much benefit! There is almost no difference in the mean for the two tests, and the difference in the median is well within one standard deviation.  Are we sure this worked?  Let's check the details:

With the CDN in place we get external IP addresses for every image (even though the location is an ambiguous "United States"):



With the scripted test we are hitting the origin in Boston, MA for every image:


So the test is working as expected.  Let's look a little closer...the column on the left is "Time to First Byte" (TTFB) and the next column over is "Content Download".  With a CDN you expect TTFB to be comparable or better than your origin, since CDNs should have highly optimized servers and the first packet shouldn't have too far to go.  The content download should be MUCH faster, since the edge nodes are closer to the end user than your origin is.  As we scan down these screenshots we can see that for the CDN the content download times vary widely, from 2ms to 1500ms, and for images that are less than 10KB!  This is not great performance.  I'm also surprised that the TTFBs are so high, I would expect the first byte of an image to be served off an edge node in well under 100ms.  The origin requests are slower on the whole, but more consistent.  These two factors combine to make the difference in the two tests negligible.  

Based on this brief analysis of one URL, it looks like our CDN isn't providing a whole lot of benefit.  Granted, these images are on the smaller side, and CDNs should perform much better for heavier content, but the reality is that most of our static content is product images that are less that 10KB, so this is our primary use case.  I'm going to take this data back to our CDN and have a conversation, but in the meantime I hope you use this technique to see how your CDN is impacting your site's performance.