Five Email Testing Pitfalls to Avoid

Posted by Julia Peavy on

With the holidays fast approaching, many email marketers are gearing up for the busiest time of year. Even though time and budget may be tight, email testing needs to be a part of your email marketing plans. Without testing, marketers can be leaving a great deal of money on the table.  However, a poorly implemented testing program can actually be detrimental to your program. During the busy holiday season, watch out for these five testing pitfalls.

While potentially obvious, it is worth pointing out that email test cells need to be deployed at the same time (unless timing is the element that is being tested).  If the test cells are sent at different times, you can’t be sure the element being tested was the reason for the difference between the cells.

Interpreting Results Too Soon
Email marketers are extremely busy (especially during the holiday season). This hurried environment can lead to interpreting results too soon in the effort to get the “winning” campaign out the door.  If tests aren’t given enough time to run, you might deploy “the winner” which ends up actually being the loser. If possible, wait 48-72 hours before analyzing the results and deploying the winner. 

Not Statistically Relevant
There’s nothing more frustrating than taking the time to deploy a test and not having the test cells be statistically relevant. If you don’t have statistically relevant cells, you can’t be confident your test is a likely indication of how your overall audience would behave. As a general rule, 2000-3000 individuals per test cell is a good guideline. Also, be sure your sample is randomized across your audience.

Unintended Consequences
Make sure to evaluate all of your campaign metrics to completely understand the test results. While you might have increased clicks with a special discount trying to win-back lapsed subscribers, you might have also increased your complaints to an unacceptable level.  Your initial analysis might lead you to believe that one campaign was the winner, but a closer look at all the metrics provides a more complete picture.

Hidden Variables
Are there things outside the test environment that could be affecting the results to your email test? Direct mail, in-store offers, or special promotions on your social media sites are all things that could skew results. Do the holidays or seasonality make certain behavior more likely? Pay attention to what is going on outside of the email program to make sure you understand all the factors that could be affecting your test.

Avoiding these testing pitfalls during this holiday season can help ensure your email holidays are happy.


This post originally appeared in The Magill Report.

Popular this Month

 Video in Email: Is It Right For Your Business? (Part 1)

Video in Email: Is It Right For Your Business? (Part 1)

Video in email is nothing new. Marketers have been using some form of video...

Read More

 [New Research] Are These Hidden Metrics Harming Your Deliverability?

[New Research] Are These Hidden Metrics Harming Your Deliverability?

Reaching the inbox is not as simple as hitting send. Once a message is...

Read More

 What Job Is Your Subscriber Hiring Your Email To Do?

What Job Is Your Subscriber Hiring Your Email To Do?

Over the last 16 years, I’ve worked as a product manager, run product...

Read More

Author Image

About Julia Peavy

Julia Peavy is currently the Director of Partner Services at Return Path. In this role, Julia is responsible for supporting Return Path’s partners and helping improve the client experience through consistent, quality, and scalable services. When Julia's not helping partners and their clients, you can find her on the slopes, trying to lower her handicap, looking for bargains, or watching one of her boys' many sporting events.

Author Archive

Stay up to date

Enter your name and email address below to subscribe to our mailing list.

Your browser is out of date.
For a better Return Path experience, click a link below to get the latest version.