If you’re not familiar with the term, spam messages are annoying, unsolicited email messages (usually advertisements for bogus products). Depending on who you listen to, spam constitutes between 50 and 90 percent of all email traffic. In addition to violating the law by sending you emails that you didn’t ask for, spammers often use their messages to perpetrate fraud on the people who respond. Unless you’ve been offline for the past 15 years, you know all this and you’d never reply to a spam email. Nonetheless, this deluge of unwanted emails can overwhelm you, fill your inbox and drown out legitimate messages. Furthermore, spam messages often contain spyware, viruses and other forms of malware.
As an IT manager or accidental techie, you have two main strategies for fighting spam. First, educate yourself and your colleagues about how spam works. The message here boils down to: Be careful about when and where you divulge your email address. See Things You Can Do to Prevent Spam for more information on the education and training approach. Second, you can use technology to battle spam. The rest of this article will focus on anti-spam technologies, also known as spam filters.
How Much Spam Is Too Much?
While no anti-spam technology stops every nuisance email, you shouldn’t settle for a situation where spam affects your office’s productivity. The threshold of annoyance is different for everyone, but if more than a handful of messages slips through your filter every day, you should change the settings or consider a different solution. Don’t just rely on your own impression; ask your colleagues as well. They might receive more spam than you do, depending on how long they’ve had their email address and how careful they’ve been with it. Also, look at the security impact. If you’ve had virus outbreaks related to spam, you may need a better filter, even if the overall volume is low.
If you’re trying to make the case for a spam filter to your boss or your board (or yourself), check out Google’s Return on Investment Calculator. It gives you a rough estimate of the money and staff time you’re losing to spam each year. Bear in mind that Google has its own filtering solution that it wants to sell you.
Types of Spam Filters
As with most technologies, spam filters come in all shapes and sizes. They range from free, lightweight desktop software utilities to expensive, complicated hardware devices.
Desktop spam filters
A quick search will bring up hundreds of free and low-cost spam filters that you can install on your desktop. Email programs such as Microsoft Outlook and Mozilla Thunderbird usually have some spam filtering functionality built in. If you feel your email client is underperforming in this respect, consider a desktop spam filter such as Mailshell, SpamAssassin or SpamBayes. Mailshell is available for donation or at a discount at TechSoup Stock.
Some desktop filters are standalone programs, while others operate as plug-ins for your email client. Either way, make sure the spam filter works with your operating system and your preferred email client. Also, while desktop filters are appropriate for small offices, they don’t scale well. If you have more than a handful of full-time staff, consider working with your Internet service provider (ISP) or implementing one of the enterprise-level spam filters mentioned in the following section.
Enterprise-level spam filters
If you host your own email server, you’ll need an enterprise-level spam filter of some sort.
When you buy a software-based, enterprise-quality spam filter, you can usually install it on the email server itself or on a separate standalone server. Also, more and more filters are available as virtual appliances. Red Earth Policy Patrol is a server-based, enterprise-level filter, and qualified nonprofits can apply for a donation at TechSoup Stock for a $60 administrative fee. SpamTitan, Cloudmark and Brightmail are other examples.
A hardware-based filter will usually cost more. On the other hand, dedicated devices often scale better than other solutions, and they frequently offer a wider variety of features and controls. Examples of hardware-based filters include Barracuda Spam Firewall and SonicWALL Email Security.
Off-site hosted spam filters
Postini and MessageLabs, among others, offer an interesting managed solution. They provide spam filtering on their equipment at their data centers. All email sent to you passes through their filter first. Using this approach, you don’t have to install any software or hardware, but you still have a high degree of control over the filter settings for your organization. While you can contract with these companies directly, Internet service providers and hosting companies often provide this type of service as well.
ISP spam filters
Almost all Internet service providers implement some level of spam filtering. However, they often block only the most egregious and easily identified spam. So relying on the ISP’s filter won’t work for most organizations. Also, if you see an ad for an ISP-level spam filter, it’s probably not suited to your situation.
There are hundreds of different filtering technologies, and these strategies change all the time in response to the changing attacks used by spammers. In other words, there’s an arms race going on behind the scenes. A list of every technique would be tedious and quickly outdated. However, most of these approaches fall into four categories. Understanding these categories may help you evaluate and implement a spam filter in your organization. Bear in mind also that any spam filter you choose probably relies on more than one approach. For more detailed information, read Ten Spam-Filtering Methods Explained or Wikipedia’s article on Anti-Spam Techniques.
Content filtering (aka keyword filtering or Bayesian filtering)
Content filtering looks at a variety of factors within the message itself to decide whether it’s spam. For example, if “Viagra” appears 20 times, the message is probably spam. Most spammers are more subtle and devious than this, and content filters have to change constantly to keep up. The most effective content filters learn from experience. In other words, as administrators and end users mark some messages as spam and classify others as legitimate, the filter will determine the characteristics of each and improve its filtering techniques. Therefore, the IT administrator or end users may have to “teach” the filter for a few days or weeks after it’s first installed.
Blocking (aka blacklisting)
Certain domain names and IP addresses are so notorious for spamming that network administrators and ISPs ban all messages from them. This technique, known as blacklisting, is heavy-handed and easy to circumvent, but it requires fewer system resources, so it’s a good way to fight the less persistent spammers. In other words, blocking based on IP address or domain name is faster than performing a careful analysis of word frequency and other content-based criteria. However, choose your blacklist provider carefully. If your blacklist is poorly managed, you’ll start to lose important, legitimate messages from correspondents falsely identified as spammers. Also, your spam filter should let you override the blacklist for specific senders. Finally, the spam filter should let you switch blacklist providers without a hassle. Many vendors provide a blacklist as part of their service. If not, they can usually recommend a third-party provider.
This approach is controversial, so think carefully before you implement it. With a challenge/response approach, you or your IT department will create a list of approved senders and approved domains. Anyone outside your organization who sends you a message receives an automatic challenge email if they aren’t on the “whitelist” of approved senders. If they reply to the challenge, their original email goes through to the intended recipient. If not, the email is deleted. This approach frustrates spammers, but it also burdens responsible senders of appropriate messages. Some of these folks will ignore the challenge email and their message will never arrive.
Collaborative filtering harnesses the collective intelligence of everyone who uses a particular product or service. Every time an end user marks an email as spam, a centralized database takes note. If enough people blacklist the same message or the same sender, the filter starts to block those emails for everyone.
Evaluation Criteria for Spam Filters
The standard criteria for evaluating all technology obviously apply to spam filters as well. How expensive is it? How complicated? How much time does it take to learn? Does it integrate well with our existing infrastructure? Is it reliable? How good is the documentation, tech support and user community?
In addition to these general considerations, spam filters have their own unique characteristics. In particular, you should think about speed, convenience, precision and recall:
Speed and power
Some filtering techniques take longer than others. Some software/hardware combinations analyze incoming emails faster than others. The speed and power of your spam-filtering technology determines its ability to scale.
Convenience and flexibility
How much work do end users need to do? As mentioned, the challenge/response system puts more of a burden on the senders of legitimate email. Can end users add certain senders and domain names to the whitelist? Can they set up rules to let in certain messages and block others?
False positives (precision)
A false positive occurs when the filter blocks a legitimate email or puts it in the junk folder. False positives are especially dangerous because end users lose important messages. The average number of false positives for a spam filter should be at or close to zero.
False negatives (recall)
A false negative occurs when your spam filter fails to identify a spam email and lets it through to your inbox. A few false negatives are inevitable and don’t cause much harm, but a really high rate is obviously a problem. Soon you’re back where you were before you implemented the spam filter.
If you want to learn more about controlling this nuisance, check out TechSoup’s Spam Prevention Toolkit. You’ll find an article there to help you ensure that your bulk emails aren’t accidentally flagged as spam. A related piece describes the CAN-SPAM Act and the basic precautions you need to take to avoid violating that law. Getting Clueful: Five Things You Should Know about Fighting Spam explains why spam fighting will always require trade-offs and imperfect solutions.