Return Path’s Sample Size Calculator: Your Key to Testing

Posted by Alicia Herr 

Have you ever put together the perfect test campaign, only to run into the issue of how many people should receive it? You know you can’t send it to your whole list, but is an A/B split the right option? Look no further than Return Path’s Sample Size Calculator.

Return Path’s Sample Size Calculator takes all the guess work out of determining the percentage of subscribers who should be receiving that message. Return Path can help you find out how large your sample population should be to accurately reflect the true population of your list. All you need to do is simply input your confidence level, list size, and acceptable margin of error and SHAZAM, sample size calculated.

This allows you to avoid using equations that look like this:


When you select a confidence level, you are selecting the probability that the statistic you are seeing from the sample population falls within a certain percent of the true value. So the higher the confidence level the more accurate your statistic will be. This; however, will require a larger sample size because the closer your sample statistic is to the actual value, the closer your sample size will have to be to the actual population size (list size).

The second piece of information is easy — just enter the list size!

The last element—acceptable margin of error—allows you to choose the acceptable range of values for how close the figure needs to be to the true value. If you choose a margin of error of percent you are accepting values that are within five percent of the true value.

The output you receive from inputting the above values is the sample size. Didn’t see that coming, right? The sample size provided will produce values that are within the acceptable margin of error to the true value with a probability equivalent to the confidence level the user chose!

Below is an example of how the Sample Size Calculator can be used:
Bob wants to see how his new creative will perform with the 2000 subscribers on his list. He doesn’t want to mail to too many people until he understands how his subscribers will react. Using the Return Path Sample Size Calculator, Bob inputs a confidence level of 95 percent, list size of 2000 and an acceptable margin of error of five percent into the Sample Size Calculator. See below.


The Sample Size Calculator calculated that a sample size of 322 would be sufficient to produce values that are within five percent of the actual value the 2000 subscribers produce with a probability of 95 percent. Would you have guessed that on your own? Or would you have over sent?

Testing is very important in the world of email. You don’t want to be too stagnant and your subscribers become bored and unengaged. You also don’t want to go to the extreme and constantly change your emails and lose people who are confused by your ever changing branding. This confusion can lead to complaints which are even more detrimental to your email program. The best way to go about testing is finding the right audience to test. It’s easy to do a 50/50 test or even a 10/10/80, but if you really want to take the guesswork out of testing then the Sample Size Calculator is the way to go!

Author Image

About Alicia Herr

Alicia is an Account Coordinator with Return Path. Her goal is to foster and maintain good relationships with clients. She looks forward to providing her clients with better insight into their email marketing strategies and results. In her free time, you can find Alicia spending time with her nieces and nephew.

Author Archive