An Anti-Spam Method with SMTP Session Abort

1
An Anti-Spam Method with SMTP Session Abort

• Nariyoshi YAMAI1 Kiyohiko OKAYAMA1 Takumi SEIKE1
• Keita KAWANO1 Motonori NAKAMURA2 Shin MARUYAMA3
• 1 Okayama University, Japan
• 2 National Institute of Informatics, Japan
• 3 CO-CONV Corporation, Japan

2
Contents

• Existing anti-spam methods
• Anti-spam method with SMTP session abort
• Implementation and evaluation of prototype system
• Conclusions

3
Existing anti-spam methods
4
Tempfailing (1)

• Utilizes difference of MTA behavior after
  temporary error
• Legitimate MTAs
• Retry to send the temporarily failed messages
• Spam sending MTAs
• Prefer throughput
• Give up resending the temporarily failed messages

5
Tempfailing (2)
Saves triplet ( Sender IP, SMTP From, SMTP To)
First Delivery
Second delivery
retry
Sender IPSMTP FromSMTP To
MTA
Legitimate MTA
temporary error
Sender IPSMTP FromSMTP To
Spam sending MTA
temporary error
Recipients
6
Tempfailing (3)

• Problems
• RFC2821
• 4.5.4.1 Sending Strategy (excerpt)
• The sender MUST delay retrying a particular
  destination after one attempt has failed. In
  general, the retry interval SHOULD be at least 30
  minutes.

Causes large delay for legitimate mail delivery
7
Tempfailing (4)

• Problems (cont.)
• Utilizes the following triplet for retransmission
  judgment
• Sender IP
• SMTP From
• SMTP To

Rejects retries from a different MTA
8
Tempfailing (5)

• Problems (cont.)
• Rejects before receiving header/body
• Logs only the triplet (Sender IP, SMTP From,
  SMTP To)

Difficult to recover false positives
9
Distributed collaborative filter
MTA
Spam sending MTA
Only messages already read by existent recipients
can be filtered out
check
not found
found
spam
register
Spam database
Recipients
10
Anti-spam method withSMTP session abort
11
Summary of known problems

• (Tempfailing) Large delay

• (Tempfailing) Retries from a different MTA
• (Tempfailing) Recovery from false positives

• (Distributed collaborative filter) only messages
  read by recipients into DB

12
Features of the proposed method

• (Tempfailing) Large delay

• Introducing two mail gateways (MGs)
• Immediate fallback to the secondary MG

• (Tempfailing) Retries from a different MTA
• (Tempfailing) Recovery from false positives

• SMTP session abort function
• Preserving header/body on first attempt
• Retransmission judgment with Message-ID or
  checksum instead of IP

• (Distributed collaborative filter) only messages
  read by recipients into DB

• Automatic registration of unresent/undeliverable
  messages
• Early registration of many spam mails

13
System layout and behavior (1)
Preserving header/body in case of false positive
Retry

header body
Primary mail gateway
Preserving header/body
Mail gateway
InsideMTA
TCP segment (RST)
SMTP session abort
header body
Check triplet (MsgID/checksum, SMTP From, SMTP
To)
Secondary mail gateway
After SMTP session to the primary MG is aborted,
a legitimate MTA usually sends the message to
the secondary MG immediately.
Reducing delay of legitimate mail delivery
Spam database
Retransmission judgment based on header(MsgID)
or body(checksum)
Organization
Recipients
14
System layout and behavior (2-1)
Unknown recipient
RCPT TO
header body
Primary mail gateway
Spam sending MTA
recipient check
InsideMTA
undeliverable
Secondary mail gateway
register
header body
Spam database
Organization
Recipients
15
System layout and behavior (2-2)
Unknown recipient
RCPT TO

header body
Primary mail gateway
Recipient check
InsideMTA
SMTP session abort
Recipient check
header body
formerly deliverable
Secondary mail gateway
RCPT TO
register
Automatic registration of unresent/undeliverable
messages
cancel
header body
Spam database
Organization
Recipients
16
User preference of abort timing (1)

• Affects network traffic and delay
• Possible options
• Accept
• No session abort
• Header
• Abort after End of Header
• Low traffic/delay
• Body
• Abort after End of Message
• Easy recovery on false positives

17
User preference of abort timing (2)
RCPT TO A RCPT TO B RCPT TO C

Primary mail gateway
header body
RCPT TO A
InsideMTA
SMTP session abort at end of message
RCPT TO B RCPT TO C
Secondary mail gateway
RCPT TO A RCPT TO B RCPT TO C
accept
header
body
Spam database
Organization
A
B
C
18
Implementation and evaluation of prototype system
19
Prototype system implementation

• Platform
• FreeBSD with sendmail DCC
• SMTP session abort function
• An external program using ipfw
• Retransmission judgment
• (Message-ID, SMTP From, SMTP To)

20
First operation test (1)

• Objectives
• Performance evaluation of blocking/filtering
• Test domains
• Some sub-domains in okayama-u.ac.jp
• Already obsolete five years before
• To be removed in one month
• Some legitimate mails were possibly sent to these
  domains
• Test period
• Seven days from Jan. 29 to Feb. 5th, 2006

21
First operation test (2)

• Result

Number of mails processed 54,719
Number of mails blocked 44,303
Number of mails received 10,416
Number of mails filtered out by DCC 2,180

• 81 (44303/54719) of mails processed were blocked
  by SMTP session abort
• 20 (2180/10416) of mails received were filtered
  out by DCC
• NB we counted both legitimate mails and spam
  mails.

22
Second operation test (1)

• Objectives
• Comparison with conventional tempfailing as for
  processing of legitimate mails
• Test domain
• New sub-domain dedicated for this test
• Only 1 IP address available
• Two MGs have the same IP address
• Usual in small companies in Japan

23
Second operation test (2)
All messages even from gmail.com were accepted
without whitelist
Small delays of mail delivery from many domains

• Result

Some domains using qmail still had large delays
Domain (service) MTA Resend Different MTA Min. interval Min. interval
cc.okayama-u.ac.jp (Univ.) sendmail YES NO 0 (sec)
nifty.com (ISP) sendmail YES NO 1
listbox.com (ML) postfix YES NO 1
yahoo.com (free mail) ? YES NO 10
gmail.com (free mail) ? YES YES 385
aol.com (free mail) ? YES NO 6
hotmail.com (free mail) SMTPSVC YES NO 6
yahoogroups.jp (free ML) ? YES NO 1
freeml.com (free ML) qmail YES NO 399
mag2.com (mail magazine) qmail YES NO 3264
trashmail.net (anonymous mail) postfix YES NO 6
24
Possible false positives

• Messages without Message-ID
• Use Date field (mandatory), or
• Use the checksum of the body
• MTAs without retransmission
• Can recover lost headers/bodies easily
• Find such MTAs and register them into whitelist
• MTAs changing SMTP From address
• Use (Message-ID, SMTP To) without SMTP From for
  retransmission judgment

25
Conclusions
26
Conclusions

• Combination of three functions
• Tempfailing
• Distributed Collaborative filter
• SMPT session abort
• Reduces the drawbacks of existing two methods
• Future works
• Long term actual performance evaluation
• Combination with on-the-fly filters

27
Questions ?Please speak slowly and clearly