Monday, September 5, 2011

Sharing Performance Data

I am very impressed with Etsy's decision to share some of their performance data.  This is rarely done in the WPO world, companies regularly share data like "improving landing page performance by 2.2 seconds increased download conversions by 15.4%" but you almost never see absolute numbers.

Why don't companies share hard numbers?

The obvious answer is that they want to keep whatever competitive advantage they have in the speed department.  If a site starts sharing its performance data then competing sites have a benchmark to shoot for.  Other companies could also use that data as an advertising ploy, saying things like "don't shop at Etsy, our site is twice as fast!".  On top of that, performance numbers are somewhat complicated to share and compare.  As I talk about in my post on SLA's, the numbers can change dramatically depending on how you measure them.  Etsy has done an okay job of explaining their methodology, they specified that they are talking about server side performance and they gave average and 90th percentile.  Since server side performance is location and network independent there isn't a ton of variability based on measuring tools.  If Etsy starts sharing full page load times I hope they will give a bit more information about how the tools they are using to prevent people from making apples to oranges comparisons.


Why did Etsy decide to share these numbers?

This is a better question for Etsy, but I think it has to do with hiring.  Etsy open sources a ton of their projects, and they have an active tech blog.  They are a pretty small company (although growing) but their engineering group has a ton of street cred, especially in the WPO space.  This helps Etsy attract quality engineers that want to work on exciting projects, something that is very difficult to do these days.  At Wayfair (my employer) this is a problem that we deal with all the time.  It seems like every tech company in Boston is hiring, and you have to find a way to differentiate yourself from the competition.  At the end of the day I think engineers want to work on cool projects that push the boundaries of what is possible on the web.  We do a lot of that kind of work at Wayfair, but we haven't promoted ourselves nearly as well as Etsy has so far (this will be changing!).  Sharing performance data, releasing performance tools to the public, and speaking at performance conferences have made Etsy a leader in the space and an attractive employer.

Is Etsy.com Fast?

Overall I think these numbers are decent, but not spectacular.  I hope to be sharing some Wayfair numbers soon as a comparison, but I can say that our server side performance is slightly better than Etsy's based on the numbers they have shared.  From reading their post it sounds like they do a lot of personalization on each page, and they handle this on the server side while they are building the base page.  We factor out most of this work and make Ajax calls to get customer specific data, which helps us generate the base page more quickly.  We also have a different model, and the vast majority of our users are not signed in, so we wouldn't want to focus most of our energy on solving for that case.  As Seth mentions in the post their homepage performance is less than steller, but they plan on improving it in the near future.  The important thing here is that they DID share the data, knowing that it's not the best in the world.  Etsy isn't trying to come out and say "look at how fast we are!" they are simply sharing information with the community and with their customers, and then explaining where they are going to target improvements.  This is very commendable and I hope more companies follow suit and release their own performance data.