Whitelistin’ Ain’t Easy

Posted by J.D. Falk on

Whitelists exist because spam filters exist. They are the exception policy, the safety valve. But beyond that simple truism, there are a lot of differences.

Because there’s so much spam, filters have to rely on patterns derived from similarities between known spam messages. When a message matches the pattern, the filter notices and does something: reject it, put it in a spam folder, et cetera. Messages that don’t match the pattern sail on through.

Similarly, if the message’s source — usually tracked by IP address — matches the pattern, all messages from that source are noticed by the filter. This could be as specific as a single IP address, or could be a range of IP addresses. When a filter’s pattern is broad, it catches a lot of spam. But it may also catch some non-spam messages; this is what’s called a “false positive.” To avoid those, you could (and probably will) improve the filters over time — but by the time you find out, the damage is already done. In the meantime, you need a whitelist.

Most mail system administrators will whitelist their own network infrastructure; it’s under their control (or under the control of someone nearby), so if any problems come up they can fix them. Also, it’s generally a bad idea to block mail from your boss.

Then you’ll want to whitelist companies and organizations you and your users frequently interact with. Do a quick mental inventory: how many is that? Did you remember your payroll company, your health insurance benefits broker, your local pizza delivery joint? What about the company your local pizza joint outsourced their email to — how many other companies do does that company send for? Do they all deserve a free pass around your spam filters?

Pretty soon, managing exceptions to your filters becomes more complicated, more time-consuming, than managing the filters in the first place. And then the phone rings: some company you’ve never heard of, asking to be whitelisted so they can send their newsletter to a VP you’ve never even met — but you’ve heard she thinks it’s easy (and fun) to replace technical staff like you. Or maybe you work for an ISP, and the frat boy on the phone insists that hundreds of your users are just begging for this email. You can’t call every single user in the middle of the night to ask if that’s true. How do you decide?

First, is it a real company? That’s harder to determine than you might think. It’s easy to pretend to be a real company online — the bad guys do it a thousand times a day. Domain names registered with a false name, address, and credit card, or simply paying extra to hide their information. Web sites that look just like a real company, but aren’t. Hiring people to receive money and packages on their behalf, giving the false impression of a real physical “brick & mortar” presence. It goes on and on.

Next, obviously: are they a spammer? Spamhaus’s Register of Known Spam Operations (ROKSO) tracks the most egregious entities, digging deeper into their operations than anyone except possibly law enforcement. But spammers can read ROKSO, too, to avoid using the aliases or front companies that Spamhaus knows about. And there are thousands of companies (real or fake) which aren’t quite bad enough to land on ROKSO, but still send a lot of unwanted email.

You could also check some of the many public DNS blacklists, but be warned that some list a lot more than just spam sources. Many of these lists are run by volunteers, with widely varying levels of accuracy.

Then there’s the infrastructure checks. Does the reverse DNS accurately reveal what kind of machine the IP address is assigned to? Often, no. Is there a valid SPF record? Can’t be absolutely certain without seeing mail traffic from the other IPs in the record. Are role accounts like abuse@ and postmaster@ functional and staffed? Takes some work to find out.

If they send ads, or newsletters, or host discussion lists, does the unsubscribe link work? Is there an unsubscribe link? You’ll have to subscribe to find out; while you’re at it you can read their privacy policy (often written in dense legalese), and find out if they’ll admit to selling your address to any “affiliates.”

All that is only the beginning. Sending practices change over time. Bad guys have been known to pretend to be good long enough to get past the application process, then start blasting away until they’re kicked off the whitelist.

So for ongoing maintenance, you’ll need to develop a system to track if your users are complaining about the mail they’re receiving…and set up some spam traps to see if this sender is “guessing” other addresses at your domain, or buying bad lists…and subscribe seed accounts to see if they’re sending different content after being whitelisted…and, wouldn’t it be useful to know if your peers, the mail admins managing other networks, have been seeing any problems?

Who will do all this work? It’s far too much for one person, particularly when your real job is to keep the servers from melting down under the load of spam. There are a few experienced people on the market right now, but that’s a rare occurrence and won’t last long.

Oh, and what happens if your research says no, don’t whitelist this entity — and then they threaten to sue if you don’t? You’ll probably win the suit, but it will be very expensive and extremely annoying along the way.

Microsoft, known throughout the industry for going their own way, employing a bigger legal team than many sovereign nations, looked at this problem and decided they needed help. MSN Hotmail (and now Windows Live Mail) uses Return Path’s Certified program as a whitelist — and has since Certified’s ancestor, the Bonded Sender Program, was introduced in 2002.

And they aren’t alone. Yahoo!, Time Warner Cable/Road Runner, Cox, Rogers, Comcast, Sympatico, BT Internet, Eleven GMBH, GoDaddy, USA.net, Rackspace, Pair Networks, Cloudmark, SpamAssassin, and many more have chosen to use Return Path’s Certified or Safe lists to help fine-tune their filtering policies, letting legitimate mail in while they work at keeping the bad stuff out.

Any of these companies and organizations could, with enough time and effort and expense, create a whitelisting program as good as ours. So could you, if you had access to the vast swaths of reputation data underlying the Sender Score, plus a few data sources only used in Certified and Safe. You could spend years building tools and hiring people to perform every check mentioned above, and more besides.

But is it really worth all that effort to go it alone, making email even more of a cost center, when we’re here to help?

Author Image

About J.D. Falk

Author Archive

Your browser is out of date.

For a better Return Path experience, click a link below to get the latest version.