Benchmarks: Is One Ever Enough?

The term “benchmark” is derived from the 19th-century practice of marking stones to indicate where “benches,” a type of bracket for mounting land survey equipment, should be placed[1].  Benchmarks allowed land surveyors to locate their equipment in exactly the same place from survey to survey and thereby ensure the accuracy of their work.

From those simple origins, benchmarking has become ubiquitous in today’s data-saturated business environment.  With more and more data at their fingertips, marketers are increasingly pressed to find ways of objectively evaluating (and justifying) their marketing campaigns.  Benchmarking provides ready answers to persistent “how are we doing?” questions from stakeholders, and is particularly useful for email marketing.

However, one of the biggest benefits of email benchmarks—their universality—can also be their greatest drawback.  Not unlike their 19th-century namesake, today’s email benchmarks are intended to provide a bias-free means of comparing email campaigns using large data sets from a wide range of brands and campaigns.  But it may not always make sense to assess the effectiveness of one’s own email marketing program against publicly available benchmarks, at least not without knowing more about the source and composition of the underlying data.  If the brands and email programs comprising a benchmark study don’t look a lot like my own brand and program, does it make sense to evaluate my own email marketing success based their aggregated results? 

This challenge was somewhat mitigated in recent years when benchmark studies started breaking out findings by industry vertical. But for one vertical that makes heavy use of email, the travel industry, significant challenges persist.  Travel is a large, fragmented vertical that really is a constellation of industries (including airlines, hotels, car rentals, online travel agencies, and others).  And travel companies that are actively using email to drive sales run the gamut from boutique shop to large multinational corporation.  Almost by definition, therefore, email marketing benchmarks for the travel industry will have widely varying degrees of relevance, depending who is comparing themselves to whom.  Without knowing the specific brands whose data is being used to create a benchmark, it is very difficult for travel marketers to accurately assess their own performance.

The findings of a recent benchmark study provide a useful example.  In April Silverpop released its annual (and very good) “2014 Email Marketing Metrics Benchmark Study”.  Unlike other leading email benchmark studies, Silverpop breaks out data by country/region, industry, and type of message (manual vs automated, and transactional vs. non-transactional).  In addition, it takes the extra step of providing mean, median, and top and bottom quartiles for data in all categories, which adds a useful layer of additional detail to the findings.

The travel industry fared particularly poorly relative to other industries in Silverpop’s study, with metrics that were the lowest or nearly the lowest across the board.  And while Silverpop may not the be the largest of top-tier Email Service Providers (ESPs), its client roster does include some well-known travel brands, including two international airlines and a global provider of travel services.  One can therefore safely assume that at least some of Silverpop’s travel benchmark data is based on the performance of large travel brands.

So should travel marketers be worried about the results of Silverpop’s study?  Are consumers no longer interested in receiving travel offers and information via email?  Or should marketers be dancing in the streets because their own programs are significantly outperforming Silverpop’s benchmarks?

As is so often the case, the answer is somewhere in the middle.

Return Path’s Inbox Insight consumer data stream provides another perspective on the efficacy of email for promoting travel. Comprising millions of active consumer inboxes, the Inbox Insight data stream provides a view into how often real-life consumers are actually reading (or not reading) emails from a wide range of brands, and how often they are reporting messages as spam.

The table below compares 2013 Inbox Insight Read Rate data for 21 bellwether travel brands to Silverpop’s Open Rate benchmarks, breaking out data in the same manner as Silverpop’s study.  Inbox Insight subscribers are reading travel emails at significantly higher rates than recipients of travel emails in Silverpop’s study, with a narrower spread between quartiles for all brands.

Does this discrepancy mean that there is a problem with either data set?  No, not at all.  What it does suggest is that email marketers need to be careful about one-size-fits-all approaches to using benchmarks for evaluating performance.

Email metrics are like streetlights on a winding road.  If one uses only one data point, it is like trying to figure out where that road is going using a single streetlight.  Just as multiple streetlights are needed to see the entire stretch of pavement at night, email metrics and benchmarks have to be used in unison to make holistic sense of a program’s performance.  That is why email marketers do themselves a disservice by seeking a single benchmark to evaluate their programs’ performance.  

There are a number of possible explanations for the difference between the findings summarized in the table:

  • The companies represented by the Inbox Insight data are some of the largest brands in the travel industry, and generally send high volumes of email.  Without knowing the specific abrands included in Silverpop’s study, this could be a case of comparing apples to oranges.  As mentioned earlier, one of the drawbacks of benchmark studies is that the brands comprising the benchmark are generally not provided by the publishers (and probably cannot be provided for privacy reasons), so it is next to impossible to figure out how similar or dissimilar the benchmarked companies are to one’s own.
  • Inbox Insight data is gathered from the recipients’—rather than the senders’—perspective, and the table data includes booking confirmations and rewards statements that typically have much higher response rates than marketing messages.  (Return Path does have the ability to break out data by message type, so look for additional details in a future post.)  Travel companies often maintain their own, separate IPs for their transactional messages, so Silverpop’s data may not include those types of messages, and may not be benefitting from their higher open rates.
  • Consumers typically subscribe their personal email addresses to receive travel marketing messages, and the Inbox Insight data stream solely comprises addresses at AOL, Gmail, and Yahoo!  It is possible that Silverpop’s lower open rates reflect the fact that some recipients are receiving travel email at their work email addresses, where travel messages are a lower priority for readers during the workday.

The takeaway here is that email marketers need to get more sophisticated and discerning about how we use email benchmarks.  The Silverpop benchmark study is excellent and provides a lot of useful information, but Silverpop can only provide the numbers within the confines of its protecting its clients’ confidentiality.  We as marketers need to get in the habit of looking at more than one data set to develop benchmarks that make sense for our specific needs and circumstances.  If only things were as simple today as they were for 19th-century land surveyors!