UPDATE: See the latest DNSBL Safety Report for current recommendations.

SpamTips.org occasionally looks at the results of Spamassassin’s nightly masscheck at RuleQA in order to analyze the performance and safety of add-on DNSBL’s. It is vitally important to know how a DNSBL is performing before deciding if it is a good idea to use it.  Many of the below DNSBL’s were tested because they indicated strong performance in other comparisons. Our analysis demonstrates that raw detection numbers can be misleading, as ham safety ratings and overlaps with other rules must be taken into consideration.

Today’s report examines Hostkarma, SpamEatingMonkey, Tiopan, UCEProtect, Mailspike, and Nix Spam and Lashback UBL.  Recommended scores below are what I personally use in production.

Hostkarma: RCVD_IN_HOSTKARMA_BL (Improved)
This week’s results indicate sustained improvement in ham safety when compared to previous months.   The nearest production DNSBL this rule overlaps with is PSBL at 72% which indicates a reasonable level of independence.

Recommendation: Cautiously recommending use at 0.5 points.  We need to see sustained safety over a few months in order to recommend anything higher.

SpamEatingMonkey RCVD_IN_SEMBLACK (Caution)
This week’s results indicate very good recent safety performance, perhaps the safest we’ve ever seen this list.  SEMBLACK has shown poor safety over the past year so it will be interesting to see if it can sustain this improvement over the ensuing months.

Unfortunately safety alone does not make using this list a good idea. SEMBLACK is showing a high level of redundancy with the high scoring production rule RCVD_IN_PBL with an 91% overlap.  PBL scores so high that it could be dangerous to pile on additional score with a very similarly performing list.

Recommendation: Due to overlap with PBL I do not recommend a high score on this rule.  Perhaps 0.3 points is reasonable for now.

MailSpike: RCVD_IN_MSPIKE_BL
MSPIKE is performing a bit worse in this week’s results, but we believe this is a temporary problem with our test data. Last week’s results show typical excellent performance and safety of MSPIKE_BL. Nearest high scoring overlap is RCVD_IN_PSBL at ~66% which further confirms the uniqueness and worth of this DNSBL.

Recommendation: I highly recommend following their instructions to setup their rules. You have the choice of the simple RCVD_IN_MSPIKE_BL which works just fine, or if you prefer MSPIKE_L3, L4, L5 and Z are the components that combine to BL where you can assign more fine-grained scores. I personally recommend staying below 2.5 points for any DNSBL rule, and use a maximum of 2.1 points for MSPIKE_BL for now.

UCEProtect Level 1 (Need More Info)
This is a new entrant to Spamassassin’s weekly masscheck. Results indicate a moderate performer but its safety rating indicates need for improvement. “set 0, broken down by message age in weeks” suggests that this list operates a spam trap with ~1 week expiration.  Overlaps indicate similar data sources to the other trap-based DNSBL’s. The nearest production rule being PSBL at 73%, but 77% overlap with MSPIKE_BL and 88% overlap with HOSTKARMA_BL.

Recommendation: Watch for a few months. If its safety were better we might recommend a low score, except the high overlap with other trap-based lists (HOSTKARMA_BL especially) might make this rule redundant for Spamassassin.

Nix Spam: RCVD IN_NIX_SPAM (Redundant, Unsafe)
NiX Spam is operated by German media outlet Heise.de.  It is a bit unusual compared to other DNSBL’s in that it only lists IP addresses for 12 hours.  For this reason we are having a difficult time measuring its performance using the usual masscheck mechanisms. Reportedly it works best for European mail.

While RuleQA is unable to give any statistics, I can report results from my own servers where it caught ~50% of my spam during the past month. Unfortunately, it also caught 1.8% of my ham during that period.  Its failures seem consistent: some legitimate Japanese companies, prominent American subscriptions and various senders in the IADB whitelist.  To make matters worse, it is overlapping with high scoring RCVD_IN_PBL ~85% of the time.

Recommendation: Due to poor safety and largely redundant performance to PBL, this is a strong recommendation to avoid using this DNSBL.

RCVD_IN_TIOPAN_BL (Avoid)
While we are unable to give exact numbers due to limitations in masscheck, it is evident from Tiopan’s results that this is a very aggressive DNSBL. While it caught ~81-87% of spam, also caught  somewhere between 1.6% and 17% of ham.  None of the other DNSBL’s come anywhere close to this scary level of false positives.  Further worrisome is the overlap of PBL at 78%.

Recommendation: Avoid.  Extremely poor safety rating combined with high PBL redundancy makes this list very dangerous to Spamassassin.


Lashback UBL (Service Failure)
The last time we tested Lashback’s UBL in masscheck was during late 2009.  It performed very poorly in that test with 7.9% spam and 2.3% ham detected.  I attempted to test it again locally this past week but it was a complete failure, with their DNS servers responding anywhere between zero and 12 seconds.  Most DNS queries used by spamassassin respond in less than 0.3 seconds.  This can delay your spamassassin filter by several seconds per message before it hits the forced DNS timeout.  The weekly statistics at Intra2Net further indicates that this list suffers from reliability issues.

I only mention the problems of UBL here because I noticed a few people recommending it as a custom rule. This further underscores the importance of testing.