Complement Set Filtering (CSF) is a method for filtering unsolicited bulk email (UBE or spam) The technique utilizes at least two email accounts: the primary account where spam and non-spam is received and secondary accounts that receive only spam. CSF calculates the set theoretic difference between the primary and secondary email sets (email accounts) and identifies email messages contained in both sets.

Implementation

CSF is implemented by comparing message content in a UBE account (separate mailbox or alias) with the message content in a primary account. By definition, messages contained in the UBE account are spam so messages in the primary account that are substantially similar to messages in the UBE account are also spam. When the same message is found in both the primary account and the UBE account, it is deleted from the primary account.

The UBE account is established by creating a mailbox (or alias) incorporating a common first name (to help spammers guess the address) and the domain of the primary account, then exposing the UBE account to the internet. For example, if the primary mailbox is johnm@domain.com, the UBE account might be john@domain.com (see diagram below). After the UBE mailbox is set up, the email address is given to spammers by posting it to message boards, portal groups, “Who Is” listings, ecommerce sites and Usenet.

CSF works especially well in corporate environments where the domain is targeted by spammers and UBE tends to be very similar from mailbox to mailbox. Also, because CSF does not depend on characteristics of past UBE to identify current UBE it is particularly well suited for identifying UBE with new subject matter.

Advantages of CSF

Many spam-filtering techniques search for patterns and known spam subject matter in the headers and bodies of messages. Others use probabilities (Bayesian statistical methods, for example) to identify unwanted messages. CSF is effective as a stand alone filter or can be combined with other techniques.

CSF has at least three advantages over Bayesian and pattern analysis algorithms. First, CSF does not depend on content analysis other than what is required to find similarities between messages in the primary and UBE accounts. Second, CSF does not utilize scoring (word ranking) that can be circumvented with message obfuscating (V!agra instead of Viagra, for example). Third, CSF takes advantage of the fact most UBE contains identical message content, particularly messages targeted at specific corporate domains.

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia.

Bogofilter is a mail filter that classifies e-mail as spam or ham (non-spam) by a statistical analysis of the message’s header and content (body). The program is able to learn from the user’s classifications and corrections. It was originally written by Eric S. Raymond, and is now maintained together with a group of contributors including but not limited to Adrian Otto, Matthias Andree, Matt Martini and David Relson.

The statistical technique used is known as Bayesian filtering and its use for spam was first described by Paul Graham in his article A Plan For Spam. Gary Robinson, in his weblog Rants, suggests some refinements for improved discrimination between spam and ham. Bogofilter’s primary algorithm uses the f(w) parameter and the Fisher inverse chi-square technique that he describes.

Bogofilter is run by an MDA script to classify an incoming message as spam or ham (using wordlists stored by BerkeleyDB). Bogofilter provides processing for plain text and HTML. It supports multi-part MIME message with decoding of base64, quoted-printable, and uuencoded text and ignores attachments, such as images.

Bogofilter is written in C, and runs on Linux, FreeBSD, Solaris, Mac OS X, HP-UX, AIX and other platforms.

Links

• Official homepage
• freshmeat project page

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia.

Bayesian spam filtering is the process of using Bayesian statistical methods to classify documents into categories.

Bayesian filtering was proposed by Sahami et al. (1998) and gained attention in 2002 when it was described in the paper A Plan for Spam by Paul Graham. Since then it has become a popular mechanism to distinguish illegitimate spam email from legitimate email. Many modern mail programs such as Mozilla Thunderbird implement Bayesian spam filtering. Server-side email filters, such as SpamAssassin and ASSP, make use of Bayesian spam filtering techniques, and the functionality is sometimes embedded within mail server software itself.

Advantages

The advantage of Bayesian spam filtering is that it can be trained on a per-user basis.

The spam that a user receives is often related to the online user’s activities. For example, a user may have been subscribed to an online newsletter that the user considers to be spam. This online newsletter is likely to contain words that are common to all newsletters, such as the name of the newsletter and its originating email address. A Bayesian spam filter will eventually assign a higher probability based on the user’s specific patterns.

The legitimate e-mails a user receives will be tend to be different. For example, in a corporate environment, the company name and the names of clients or customers will be mentioned often. The filter will assign a lower spam probability to emails containing those names.

The word probabilities are unique to each user and can evolve over time with corrective training whenever the filter incorrectly classifies an email. As a result, Bayesian spam filtering accuracy after training is often superior to pre-defined rules.

It can perform particular well in avoiding false negatives, where legitimate email is incorrectly classified as spam. For example, if the email contains the word “Nigeria”, which frequently appeared in a long spam campaign, a pre-defined rules filter might reject it outright. A Bayesian filter would mark the word “Nigeria” as a probable spam word, but would take into account other important words that usually indicate legitimate e-mail. For example, the name of a spouse may strongly indicate the e-mail is not spam, which could overcome the use of the “Nigeria.”

Some spam filters combine the results of both Bayesian spam filtering and pre-defined rules resulting in even higher filtering accuracy. Recent spammer tactics include insertion of random innocuous words that are not normally associated with spam, thereby decreasing the email’s spam score, making it more likely to slip past a Bayesian spam filter.

Links

• Guide to Bayesian spam filters: part 1, part 2.

References

• (Sahami et al., 1998): M. Sahami, S. Dumais, D. Heckerman, E. Horvitz: A Bayesian approach to filtering junk e-mail, AAAI’98 Workshop on Learning for Text Categorization, 1998.

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia.

End users can protect themselves from the brunt of spam’s impact in numerous ways.

Preventing Address Harvesting

Preventing spammers from obtaining your email address doesn’t really solve the spam problem, any more than avoiding all but lowest crime areas of a city solves crime. Many people cannot hide their email addresses and most people want to meet new people via email. They just don’t want the flood of spam. It may, however, reduce the amount of spam that you receive.

One way that spammers obtain email addresses to target is to trawl the Web and Usenet for strings which look like addresses, using a spambot. Contact forms and address munging are good ways to prevent email addresses from appearing on these forums. If the spammers can’t find the address, the address won’t get spam.

There are other ways that spammers can get addresses such as “dictionary attacks” in which the spammer generates a number of likely-to-exist addresses out of names and common words. For instance, if there is someone with the address adam@example.com, where ‘example.com’ is a popular ISP or mail provider, it is likely that he frequently receives spam.

Address munging

Posting anonymously, or with an entirely faked name and address, is one way to avoid this “address harvesting”, but users should ensure that the faked address is not valid. Users who want to receive legitimate email regarding their posts or Web sites can alter their addresses in some way that humans can figure out but spammers haven’t (yet). For instance, joe@example.net might post as joeNOS@PAM.example.net, or display his email address as an image instead of text. This is called address munging, from the jargon word “mung” meaning to break.

Contact Forms

Contact forms allow users to send email by filling out forms in a web browser. The web server takes the form data and forwards it to an email address. The user (and therefore the spam harvester) never sees the email address. Contact forms have the drawback that they require a website that supports server side scripts. They are also inconvenient to the message sender as he is not able to use his preferred e-mail client. Finally if the software used to run the contact forms is buggy or badly designed they can become spam tools in their own right.

Disposable e-mail addresses

Many email users sometimes need to give an address to a site without complete assurance that the site will not spam, or leak the address to spammers. One way to mitigate the risk of spam from such sites is to provide a disposable email address — a temporary address which forwards email to your real account, but which you can disable or abandon whenever you see fit.

A number of services provide disposable address forwarding. Addresses can be manually disabled, can expire after a given time interval, or can expire after a certain number of messages have been forwarded. Some of these services allow easier creation of disposable addresses via various techniques.

Defeating Web bugs and JavaScript

Many modern mail programs incorporate Web browser functionality, such as the display of HTML, URLs, and images. This can easily expose the user to pornographic or otherwise offensive images in spam. In addition, spam written in HTML can contain JavaScript programs to direct the user’s Web browser to an advertised page, or to make the spam message difficult or impossible to close or delete. In some cases, spam messages have contained attacks upon security vulnerabilities in the HTML renderer, using these holes to install spyware. (Some computer viruses are borne by the same mechanisms.) Also, the HTML can be used to signal whether a spam message is actually read and seen by a user.

Users can defend against these methods by using mail clients which do not automatically display HTML, images or attachments, or by configuring their clients not to display these by default.

Avoiding responding to spam

It is well established that some spammers regard responses to their messages — even responses which say “Don’t spam me” — as confirmation that an email address refers validly to a reader. Likewise, many spam messages contain Web links or addresses which the user is directed to follow to be removed from the spammer’s mailing list.

In several cases, spam-fighters have tested these links and addresses and confirmed that they do not lead to the recipient address’s removal — if anything, they lead to more spam.

In late 2003, the USA FTC launched a public relations campaign to encourage email users to simply never respond to a spam email — ever. This campaign stemmed from the tendency of casual email users to reply to spam, in order to complain and request the spammer to cease sending spam.

Perhaps more significantly, since the sender address fields borne by spam messages are almost always forged, a reply to a spam message is likely to reach an innocent third party if it reaches anyone at all.

In Usenet, it is widely considered even more important to avoid responding to spam. Many ISPs have software that seeks out and destroys duplicate messages. Someone may see a spam and respond to it before it is cancelled by their server, which can have the effect of reposting the spammer’s spam for them; since it is not just a duplicate, this reposted copy will last longer.

Reporting spam

The majority of ISPs explicitly forbid their users from spamming, and eject from their service users who are found to have spammed. Tracking down a spammer’s ISP and reporting the offense often leads to the spammer’s service being terminated. Unfortunately, it can be difficult to track down the spammer — and while there are some online tools to assist, they are not always accurate. Also occasionally spammers own their own netblocks. In this case the abuse contact for the netblock can be the spammer itself and can confirm your address as live.

Examples of these online tools are SpamCop, Network Abuse Clearinghouse and Blue Frog. These provide automated or semi-automated means to report spam to ISPs. Some spam-fighters regard them as inaccurate compared to what an expert in the email system can do; however, most email users are not experts.

Consumers may also forward “unwanted or deceptive spam” to an email address (spam@uce.gov ) maintained by the FTC. The database so collected is used to prosecute perpetrators of various types of scam or deceptive advertising.

Defense against email worms

In the past several years, scores of worm programs have used email systems as a conduit for infection. The worm program transmits itself in an email message, usually as a MIME attachment. In order to infect a computer, the executable worm attachment must be opened. In almost all cases, this means the user must click on the attachment. The worm also requires a software environment compatible with its programming.

Email users can defend against worms in a number of ways, including:

• Avoiding email client software which supports executable attachments. The most frequently-targeted client software for email worms is Microsoft Outlook and Outlook Express, both of which can easily be made to open executable attachments. However, other Windows-based email software is not immune to worms.
• Using an operating system which does not provide an environment compatible with present worms. Essentially all current email worms affect only the Microsoft Windows operating system. They cannot execute on Macintosh, Unix, GNU/Linux, or other operating systems. In some cases, it is conceivable that a worm could be written for one of these systems; however, various security features militate against it.
• Using up-to-date anti-virus software to detect incoming worms and quarantine or delete them before they can take effect.
• Being skeptical of unsolicited email attachments. Since worms and other email-borne malware arrive in this form, some email users simply refuse to open attachments that the sender has not given them advance notice of.

This article is licensed under the GNU Free Documentation License. It uses material from the Wikipedia.

The continuing increase in spam has resulted in rapid growth in the use of spam filter programs: software designed to examine incoming email and separate spam emails from genuine email messages intended for the user.

Unwanted e-mail can be filtered at the desktop, the network email server/email gateway, the Internet Service Provider’s email gateway, or all three locations. While network managers and ISPs can choose hardened email security appliances, services or software designed to interdict both spam and viruses, desktop users are frequently limited to a software-based solution.

A number of commercial spam filtering programs exist and are readily available, but many freeware and shareware spam filters are also available for easy downloading and installation. Spam filters are currently included as standard features in nearly every available email client, though the quality of these built-in filters can be low; for some users, this may necessitate the use of a higher quality filtering solution.

This guide is licensed under the GNU Free Documentation License. It uses material from the Wikipedia.

E-mail has become the subject of much abuse, in the form of both spamming and E-mail worm programs. Both of these flood the in-boxes of E-mail users with junk E-mails, wasting their time and money, and often carrying offensive, fraudulent, or damaging content. This links help in the efforts to stop E-mail abuse and ensure that E-mail continues to be usable in the face of these threats.

Links

• BlackList checker Check if the domain is blacklisted by Open Relay DataBase (ORDB) or Distributed Sender Blackhole List (DSBL).
• Fight Spam on the Internet!
• Coalition Against Unsolicited Commercial Email
• Spamfo.co.uk Covers the latest news on junk email, scams, fraud, legal aspects and reviews of software and services out there to help you reduce the ‘spam’.
• California lawyer who sues spammers
• Address Munging FAQ: Spam-Blocking Your Email Address
• Challenge/Response at the SMTP Level: It may be possible to implement a challenge/response SPAM protection system using custom SMTP Delivery Status Notifications (DSNs) that include an HTML confirmation hyperlink.
• Countering Spam by Using Ham Passwords (Email Passwords)
• “Countering Spam with Ham-Authenticated Email and the Guarded Email Protocol” describes a challenge/response system
• [8] A GPL’ed minimal CAPTCHA C/R system.
• Spam-proofing the mail system; Linux Weekly News; December 17, 2003.
• hide email addresses on web sites with JavaScript or letter encoding
• Alternative way to hide email addresses on web sites with inline CSS
• Hiding Email Addresses A comprehensive list of methods on how to hide email addresses in source code from spam bots.

“SPAM” a 19th Century Problem

In the late 19th Century Western Union allowed telegraphic messages on its network to be sent to multiple destinations. Up until the Great Depression wealthy North American residents would be deluged with nebulous investment offers. This problem never fully emerged in Europe to the degree that it did in the Americas, because telegraphy was regulated by national post offices in the European region.

History of Internet “SPAM” (1978-Present)

Although spamming has existed on the Internet since as early as 1978, the first major spamming incidents didn’t take place until the early 1990s.

Spamming began becoming a major problem at the same time that the Internet began its exponential mainstream expansion in 1993 (also known as Eternal September). More recently, Tim Roarty, using the tag tjroar, spammed across so many internet forums with such a high rate of posting that he is credited with a slow down in overall internet speed. Most major forums have disabled the ability to register as tjroar on their forum due to this problem. It is suspected taht tjroar was a collection of irc trojans and they trolled the internet to sign up for forums – fortunately the creator is unable to modify them and they only try to sign up as the user name “tjroar”.

Origin of the term “SPAM”

The term spam is widely believed to have derived from the SPAM sketch of the BBC television comedy series “Monty Python’s Flying Circus”.

The sketch features a small restaurant in which every item on the menu includes SPAM canned meat, and a chorus of Vikings drowning out all conversation with a song consisting almost entirely of the word “SPAM.”

References

• History
  • Reaction to the DEC Spam of 1978 Overview and text of the first known internet email spam.

This guide is licensed under the GNU Free Documentation License. It uses material from the Wikipedia.

Spamming remains a hot discussion topic. In fact, many online users have even suggested (though they were presumably joking) that cruel forms of capital punishment would be appropriate for spammers. In 2004, the seized Porsche of an indicted spammer was advertised on the internet; this revealed the extent of the financial rewards available to those who are willing to commit duplicitous acts online. However, some of the possible means used to stop spamming may lead to other side effects, such as increased government control over the Net, loss of privacy, barriers to free expression, and even the commercialization of e-mail.

One of the chief values favored by many long-time Internet users and experts, as well as by many members of the public, is the free exchange of ideas. Many have valued the relative anarchy of the Internet, and bridle at the idea of restrictions placed upon it. A common refrain from spam-fighters is that spamming itself abridges the historical freedom of the Internet, by attempting to force users to carry the costs of material which they would not choose.

An ongoing concern expressed by parties such as the Electronic Frontier Foundation and the ACLU has to do with so-called “stealth blocking”, a term for ISPs employing aggressive spam blocking without their users’ knowledge. These groups’ concern is that ISPs or technicians seeking to reduce spam-related costs may select tools which (either through error or design) also block non-spam e-mail from sites seen as “spam-friendly”. SPEWS is a common target of these criticisms. Few object to the existence of these tools; it is their use in filtering the mail of users who are not informed of their use which draws fire.

Some see spam-blocking tools as a threat to free expression—and laws against spamming as an untoward precedent for regulation or taxation of e-mail and the Internet at large. Even though it is possible in some jurisdictions to treat some spam as unlawful merely by applying existing laws against trespass and conversion, some laws specifically targeting spam have been proposed. In 2004, United States passed the Can Spam Act of 2003 which provided ISPs with tools to combat spam. This act allowed Yahoo! to successfully sue Eric Head, reportedly one of the biggest spammers in the world, who settled the lawsuit for several thousand U.S. dollars in June 2004. But the law is criticized by many for not being effective enough. Indeed, the law was supported by some spammers and organizations which support spamming, and opposed by many in the antispam community. Examples of effective anti-abuse laws that respect free speech rights include those in the U.S. against unsolicited faxes and phone calls, and those in Australia and a few U.S. states against spam.

Court cases

Attorney Laurence Canter was disbarred by the Supreme Court of Tennessee in 1997 for sending prodigious amounts of spam advertising his immigration law practice.

Robert Soloway lost a case in a federal court against the operator of a small Oklahoma-based Internet service provider who accused him of spamming. In another case against Soloway, U.S. Judge Ralph G. Thompson granted a motion by plaintiff Robert Braver for a default judgment and permanent injunction against him. The judgment includes a statutory damages award of $10,075,000 under Oklahoma law.

In the first successful case of its kind, Mr. Nigel Roberts from the Channel Islands won £270 against Media Logistics UK who sent junk e-mails to his personal account. [15]

This guide is licensed under the GNU Free Documentation License. It uses material from the Wikipedia.

The California legislature found that spam cost United States organizations alone more than $10 billion in 2004, including lost productivity and the additional equipment, software, and manpower needed to combat the problem.

Spam’s direct effects include the consumption of computer and network resources, and the cost in human time and attention of dismissing unwanted messages. In addition, spam has costs stemming from the kinds of spam messages sent, from the ways spammers send them, and from the arms race between spammers and those who try to stop or control spam. In addition, there are the opportunity cost of those who forgo the use of spam-afflicted systems. There are the direct costs, as well as the indirect costs borne by the victims – both those related to the spamming itself, and to other crimes that usually accompany it, such as financial theft, identity theft, data and intellectual property theft, virus and other malware infection, child pornography, fraud, and deceptive marketing.

The methods of spammers are likewise costly. Because spamming contravenes the vast majority of ISPs’ acceptable-use policies, most spammers have for many years gone to some trouble to conceal the origins of their spam. E-mail, Usenet, and instant-message spam are often sent through insecure proxy servers belonging to unwilling third parties. Spammers frequently use false names, addresses, phone numbers, and other contact information to set up “disposable” accounts at various Internet service providers. In some cases, they have used falsified or stolen credit card numbers to pay for these accounts. This allows them to quickly move from one account to the next as each one is discovered and shut down by the host ISPs.

The costs of spam also include the collateral costs of the struggle between spammers and the administrators and users of the media threatened by spamming. See [1].

Many users are bothered by spam because it impinges upon the amount of time they spend reading their e-mail. Many also find the content of spam frequently offensive, in that pornography is one of the most frequently advertised products. Spammers send their spam largely indiscriminately, so pornographic ads may show up in a work place e-mail inbox—or a child’s, the latter of which is illegal in many jurisdictions. Recently, there has been a noticeable increase in spam advertising websites that contain child pornography.

Some spammers argue that most of these costs could potentially be alleviated by having spammers reimburse ISPs and individuals for their material. There are two problems with this logic: first, the rate of reimbursement they could credibly budget is not nearly high enough to pay the direct costs; and second, the human cost (lost mail, lost time, and lost opportunities) is basically unrecoverable.

E-mail spam exemplifies a tragedy of the commons: spammers use resources (both physical and human), without bearing the entire cost of those resources. In fact, spammers commonly do not bear the cost at all. This raises the costs for everyone. In some ways spam is even a potential threat to the entire e-mail system, as operated in the past.

Since e-mail is so cheap to send, a tiny number of spammers can saturate the Internet with junk mail. Although only a tiny percentage of their targets are motivated to purchase their products (or fall victim to their scams), the low cost sometimes provides a sufficient conversion rate to keep spamming alive. Furthermore, even though spam appears not to be economically viable as a way for a reputable company to do business, it suffices for professional spammers to convince a tiny proportion of gullible advertisers that it is viable for those spammers to stay in business. Finally, new spammers go into business every day, and the low costs allow a single spammer to do a lot of harm before finally realizing that the business is not profitable.

Some companies and groups “rank” spammers; spammers who make the news are sometimes referred to by these rankings (Spamhaus’ “TOP 10 spam service ISPs”, The 10 Worst ROKSO Spammers ). The necessary secretiveness of the operations makes uncertainty about how they actually determine “how bad” a spammer is unavoidable. Also, spammers may target different networks to different extents, depending on how successful they are at attacking the target. Thus considerable resources are employed to actually measure the amount of spam generated by a single person or group. For example, victims that use common antispam hardware, software or services provide opportunities for such tracking. Nevertheless, such rankings should be taken with a grain of salt.

To better understand the cost of spam to an organization, MX Logic Email Defense has posted a cost of spam calculator on their website.

Continuously updated statistics from postini track the ebb and flow of e-mail abuse without ranking spammers.

This guide is licensed under the GNU Free Documentation License. It uses material from the Wikipedia.