I found some very interesting Advice to New Programmers that I found to be very insightful.

I don’t know if I consider myself to be a “new” programmer or not, but the advice given wasn’t the normal cynical “blame everyone else” crap that’s spewed out at Slashdot and the like. Probably because the guy is a proud Microsoft employee, which is equivalent to a some sort of demon-witch in the minds of many Slashdot readers.

I’ve used Akismet for some time now, and it has worked great. It still has one important downside: it requires interaction with a third party service.

This means that it can go down, the API can change, they can start charging me, etc. Additionally, I need to be using PHP or some 3rd party blog software in order to use it. While none of these reasons really keep me up at night, it would be very useful to have an equally easy to use spam detector that does not depend on a third-party.

So. I did some research on Bayesian filters, which is an application of artificial intelligence that’s used to distiguish between 2 or more categories. The way it works is by using Bayes’s Theorum to provide a probability that some text is in each category. Typically, it’s used to tell if an email/comment is spam or not spam (2 categories). But as I discovered at this site (shaunwagner.com), it could also be used to tell if a text string is a name, address, or city/state/zip, or any amount of arbitrary categories.

The disadvantage to running my own Bayesian filter is that I have to teach and store all the “learning” myself. This is actually Akismet’s strength: leveraging a network of thousands of blog comments to provide the most accurate filter.

I may or may not have time to fiddle around with writing my own Bayesian filter, but here is how I would go about it if I were to do it in the context of blog comments:

1. Get a whole bunch of comments and spam comments. (Spam is easy enough, I get close to 100 spam comments a day. Also, I have a large database of non-spam comments that should be a good starting point).
2. Mark each of these comments “Y” for spam, “N” for not spam.
3. For each incoming comment, apply the Bayesian theorum to it to get a probability. (This is the most important part). This might be simple enough to accomplish with a SQL query (or two) and a couple calculations, but my guess is that it would be non-trivial to write without a big performance hit.
4. If the Bayes probability yields a false positive or a false negative, I’ll have to manually put it into the appropriate category. In this way, it “learns” as I “teach” it.
5. If the Bayes probability is true, then two things could happen:
   • If it’s not spam, leave it alone.
   • If it is spam, delete it (manually).

The only problem I can see with this is that the more the routine “learns,” the more spam I’ll be storing indefinitely.

Part 3 is the hardest part though, so let me expand on that a little bit. Here is the equation:

Looks a little tricky, but it’s not so bad. Here’s the general idea: For each word in the comment, find out how many times it’s in every stored spam comment and also how many times it’s in every stored legit comment. Use that to create three fractions.

For instance, “viagra”. Let’s say it was in 50/100 (X) of the spam messages, 1/100 (Y) of the legit messages, and 51/200 (Z) of all messages.

Assuming that’s the only word in the comment, let’s calculate the probability that it was each category. We’get X*Z (0.1275) and Y*Z (0.00255). This means that the message is 12% likely to be a spam and only 0.2% likely to be legit. (Another way to do it would to be just comparing the X*Z percentage to some arbitrary threshold).

You can see that this isn’t particularly difficult, but remember that this calculation would need to performed on every word in the comment. That’s where the processing time can become real trouble.