Design and Management of Email Service

1
Design and Management ofEmail Service

• ???Jason Chang
• Computer Center
• National Taiwan University
• 2005/07/08
• ????????
• jsc_at_ntu.edu.tw
• http//ccms.ntu.edu.tw/jsc

2
Outline

• A brief view of NTU email service
• Introduction to the architecture and operation
  of SMTP
• Design of a suitable email system
• Webmail solutions
• Postfix and simple configuration samples
• Spam and virus filtering
• Conclusion

3
Overview

• Electronic mail service has already evolved into
  one of the major Internet applications.
• It is not only fundamental, but also a must.
• Users may become impatient when mails were
  delayed, not to mention failed to access their
  emails.
• Imagine we meet the situation of power failure or
  cut of water supply

4
A Brief View of NTU Email Service

• Current Status
• Email statistics
• Average daily incoming 800,000
• Truly delivered 200,000(25)
• Account statistics
• Faculty/staff/official use 4200
• Student 30000
• Alumni 24500
• Account policy
• Lifelong preserved for retired faculties/staffs
  and alumni
• Server design
• Incoming relay/filter 3
• Outgoing relay 1
• Webmail 2
• Pop3 5 for students and 1 for others

5
A Brief View of NTU Email Service

• Problems we faced
• Account creation and remove
• Non-decreasing number of users
• Spam
• What we have done
• Database synchronization with campus mis systems
• Single Sign On
• Totally encrypted transmission
• Spam and virus filtering
• Ugly but work architecture for scalability concern

6
Architecture of a Simple Mail System

• Consists of the following components
• MTA - Mail transfer agent
• Sending and forwarding emails
• Server end
• MDA - Mail delivery agent
• Delivering emails to recipients mailbox
• Server end
• Pop3/Imap4 Daemons
• For users to download their mailboxs
• Server end
• MUA - Mail user agent
• Reading and composing emails
• Client end

7
Architecture of a Simple Mail System
8
Protocols Used for Mail System

• Protocols
• For computer programs to communicate with each
  other
• Similar to languages that human beings speak
• SMTP
• Simple Mail Transfer Protocol
• Too simple to provide any advanced features
• Authentication
• Authorization
• POP3
• Post Office Protocol version 3
• Simple
• IMAP4
• Internet Message Access Protocol version 4
• Fully compatible with internet message standards,
  e.g. MIME.
• Allow messages to be accessed from more than one
  computer.
• Provide support for online, offline, and
  disconnected modes.
• Multiple and share folders.

9
Mail Forwarding Between Servers
10
How to Find the Way to the Destination?

• How do we find the way tojsc_at_ntu.edu.tw?

11
DNS The Key to All Internet Services

• Query DNS server by the address part of email
  address.(jsc_at_ntu.edu.tw)
• Query for MX records
• Query for A records
• Give up!
• Not required to exist a real machine hosting the
  address.(ntu.edu.tw)
• Adding MX records to improve query performance
  and to reduce unnecessary query overhead.

12
DNS Query Results

• Precedence value of MX records
• Lower valueswith higher priority

13
DNS Query Results

• When MX record is absent, second query is
  required to issue.

14
The Use of Mail Relay Gateway

• Load sharing and balance
• Prevent the occurrence of single point of failure
• Hide your backend mail servers and prevent from
  outside attack
• Frequently used when you have M Exchange servers
• Make use of multiple network link connections
  (different ISPs)
• Spam and virus filtering
• The easiest way for administrators to release
  current bottlenecks and to leverage performance.
• Its the way how most commercial products work.

15
The Architecture of Mail Relay Gateways
16
Planning of a Modern Mail System

• This is what system administrators care.
• Scalability
• User / spool capacity
• Virtual user management
• Availability
• Multiple relay servers
• Multiple pop3 servers
• Security
• Avoid creating system accounts
• No access to shell
• Performance
• Make the right decision
• Architecture
• OS/Software

17
Features Required By Modern Mail Systems

• This is what end users care.
• Ease of use interface
• IMAP4
• Webmail
• Security
• Authentication
• Encrypted transmission
• Spam and virus filtering
• The only effective way to prevent from virus
  infection is to remind and force your users to
  install antivirus softwares.
• Stop using M systems is alternative solution.

18
Scalability of a Mail System

• Traditional way
• Additional servers with different address.
• ms1.cc.ntu.edu.tw
• ms2.cc.ntu.edu.tw
• ms3.cc.ntu.edu.tw
• Ugly and not professional
• Modern and Clever way
• Multiple backend servers
• Make use of relay gateways to dispatch and
  deliver mails
• A uniform email address for all of the users.
• user_at_ntu.edu.tw
• Virtual user management by central database

19
A Scalable Mail System Architecture
20
Improving Scalability and Availability By Adding
Multiple POP3 Servers

• POP3/IMAP4 accesses are more frequent than
  incoming mail delivery.
• Users may easily aware of the failure of pop3
  servers in comparison to smtp/relay
  servers.(inout 201)
• NFS protocol is used to share the common storage.
• Mail delivery efforts can be dealt by other
  servers.
• Totally rely on the stability of storage server.
• Single point of failure
• Could be replaced by high-end NFS servers or
  NetApp filers.

21
Architecture of Multiple POP3 Servers
22
NFS Concerns And Bottlenecks

• NFS has its own limitation.
• Do not expect NFS to scale well.
• Use gigabit network with high quality switch.
• UDP is preferred over TCP when network
  environment is local and qualified.
• Consider to apply the Maildir format
• Avoid NFS lock problem
• Severe enough in Linux and BSD systems
• Spin lock and dead lock may block further
  accesses
• Manual interference often requires

23
Make the Right Decision For the Promise of
Performance

• Capacity planning
• Reasonable system design and architecture
• Operating system
• Linux, BSD
• Always the best and wise choice.
• Solaris
• Its been called Slowlaris.
• M
• Are you kidding?

24
Make the Right Decision For the Promise of
Performance

• MTA
• Postfix
• Our suggestion!
• Sendmail
• Not scalable!
• Lack of new features.
• Not efficiency in resource management.
• Rich of documents and tutorials
• Qmail
• Termination of development.
• Exim
• Immature.
• M Exchange
• Seeking trouble for yourself?
• POP3/IMAP4
• UW-IMAP
• Courier-IMAP
• Supports Maildir format.
• Dovecot

25
Webmail The Trend of Browser Based Mail Reader

• Your determination depends on the system
  architecture.
• IMAP4 based
• Access the mailbox via IMAP4 protocol locally or
  remotely.
• Flexible, scalable
• Speed may be limited by protocol overhead and
  network transmission.
• Horde IMP project
• Locally direct access
• Access the mailbox locally, proprietary protocol.
• Could only run on local mail storage server.
• NFS could be applied when deploying multiple
  servers
• Extremely fast!! iff system not overloaded.
• Not scale well when you want to expand.
• Openwebmail
• Excellent project developed by NCKU, Taiwan
• Marvelous support to Chinese encodings
• Only support mbox format currently

26
Webmail Architecture IMAP4 based
27
Webmail Architecture Locally Direct Access
28
Postfix The Modern and Advance MTA

• http//www.postfix.org
• Developed by Wietse Venema
• Features
• Easy replacement of existing sendmail system by
  binaries with same file name and functionality .
  e.g. sendmail, newaliases
• Human readable configuration file
• Multiple small programs with limited execution
  privilege
• Multiple queues to avoid block of normal mails
• Backend database lookup table supported
• Better support for virtual host
• Virtual user system supported
• Content filtering interface
• Simple spam check/block mechanism

29
Postfix - the Big Picture

• http//www.postfix.org/big-picture.html
• Yellow ellipsoids are mail programs.
• Yellow boxes are mail queues or files.
• Blue boxes are lookup tables.

30
Postfix Installation

• Debian Linux
• apt-get install postfix-tls libsasl7
  libsasl-modules-plain courier-imap
• Redhat/Fedora Linux
• rpm -ivh postfix-2.2.x.i386.rpm
• rpm -ivh cyrus-sasl-2.1.21.i386.rpm
• BSD
• Use the ports system
• /usr/ports/mail/postfix
• /usr/ports/security/cyrus-sasl2
• /usr/ports/mail/courier-imap

31
Post Installation

• Postfix Configuration
• master.cf
• Similar to inetd.conf
• Control the behavior of small programs
• In contrast against sendmail, with one binary and
  one config file
• main.cf
• The main configuration of the mail system
• In general cases, no modification is required for
  a simple setup.
• Commands to control the postfix program
• postfix start
• postfix stop
• postfix reload

32
Test For Receiving Email

• Send an email to yourself.
• Check system logs
• /var/log/maillog
• An simple test from jason_at_dolphin.cc.ntu.edu.tw
  to jason_at_freebsd.csie.nctu.edu.tw

33
Frequently Used Configuration Options In main.cf

• myhostname mail.cc.ntu.edu.tw
• mydomain cc.ntu.edu.tw
• myorigin mydomain
• mydestination myhostname, localhost.mydomain
• On a mail domain gateway,you should also include
  mydomain.
• mynetworks_style host
• relay_domains mydestination

• home_mailbox Maildir/
• message_size_limit 20971520
• smtpd_recipient_limit 1000

34
Configure to Enable SSL

• main.cf
• smtpd_enforce_tls yes
• smtpd_use_tls yes
• smtpd_tls_cert_file /usr/local/etc/ssl/smtp.cert
  
• smtpd_tls_key_file /usr/local/etc/ssl/smtp.key
• smtpd_tls_CAfile /usr/local/etc/ssl/ntuca.crt
• smtpd_tls_loglevel 1
• smtpd_tls_received_header yes
• smtp_enforce_tls yes
• smtp_tls_cert_file smtpd_tls_cert_file
• smtp_tls_key_file smtpd_tls_key_file
• smtp_tls_CAfile smtpd_tls_CAfile
• smtp_tls_loglevel 1
• smtp_use_tls yes
• smtp_tls_note_starttls_offer yes
• tls_random_exchange_name /var/run/prng_exch
• tls_random_source dev/dev/urandom
• tls_daemon_random_source dev/dev/urandom
• master.cf

35
Test for SSL/TLS Connection

• telnet to mail server and check for welcome
  messages.

36
Test for SSL/TLS Connection

• Send an email to yourself.
• From mx1 to ccms
• Check system logs
• /var/log/maillog

37
SMTP/Authentication

• Traditional mail servers could only determine
  their relay policy based on connected ip
  addresses.
• Insufficient in face of the modern network
• DHCP, NAT
• ADSL and dialup users
• Roaming users
• Mobile, wireless
• SMTP Authentication
• Username/password authentication for each mail
  delivery.
• Reduce the chance to relay spam mails for
  anonymous.

38
Setting Up SMTP/Authentication

• Install Cyrus-SASL2 library
• It could be configured to authenticate via
  /etc/passwd, PAM, LDAP, backend databases
• Modify /usr/local/lib/sasl2/smtpd.conf
• pwcheck_method saslauthd
• mech_list plain login
• Make sure saslauthd is running.

39
Configure to Enable SMTP/Authentication

• Since the username/password negotiation is not
  encrypted. We must configure our server to accept
  incoming mail authentication only on TLS
  encrypted connections.
• main.cf
• smtpd_sasl_auth_enable yes
• smtpd_sasl_security_options noanonymous
• smtpd_tls_auth_only yes
• smtpd_recipient_restrictions reject_unknown_reci
  pient_domain, reject_non_fqdn_recipient,
  permit_sasl_authenticated, reject_unauth_destinati
  on
• master.cf
• smtps inet n - n - - smtpd -o smtpd_tls_wrappermod
  eyes
• The smtpd will listen on port 465 instead of 25.
  

40
Test For SMTP/Authentication Connection

• The username/password is required to be encoded
  in BASE64.
• The Perl MIMEBase64 module could be installed.
• /usr/ports/converters/p5-MIME-Base64
• perl -MMIMEBase64 -e 'print encode_base64("user\
  0user\0passwd")
• dGVzdDEAdGVzdDEAcGFzczE

41
Test For SMTP/Authentication Connection

• Send an email to yourself.
• Check system logs
• /var/log/maillog

42
SMTP Transaction Example
43
Email Headers

• Envelope header
• Negotiated between mail servers
• Typical tricks that SPAM mail plays
• Why do I receive mails that are not destined to
  me?
• Content Text
• Mail servers will not examine the content in
  general cases.

44
Real World Snail Mail Example

• Postman could only deliver the mail by envelope
  information.
• Sender address is not verified when email
  delivery.
• Trick of SPAM mails.
• Content may be irrelevant or conflict with
  envelope.

45
Postfix Spam Filtering Functions

• SMTP access restriction
• Refer to the previous SMTP transaction example
• http//www.postfix.org/uce.html

46
Postfix Spam Filtering Functions

• Access maps
• Similar to sendmail access map
• Control relay permission based on IP addresses
• main.cf
• smtpd_client_restrictions check_client_access
  hash/usr/local/etc/postfix/access
• Execute postmap access after update
• Dynamic IP ranges should be blocked
• http//www.twia.org.tw/SPAM-930412(news).htm

47
Postfix Spam Filtering Functions

• Realtime black list
• Use with caution
• The definition and meaning of each rbl
• Too many TW hosts are on the lists
• main.cf
• maps_rbl_domains bl.spamcop.net,
  relays.ordb.org, orbs.dorkslayers.com
• smtpd_client_restrictions reject_maps_rbl

48
Postfix Spam Filtering Functions

• SMTP access restriction
• smtpd_client_restrictions
• maps_rbl_domains bl.spamcop.net,
  relays.ordb.org, orbs.dorkslayers.com
• smtpd_client_restrictions
  reject_rbl_client relays.ordb.org
  reject_rbl_client bl.spamcop.net
  reject_maps_rbl check_client_access
  regexp/usr/local/etc/postfix/access
  reject_unauth_pipelining
  reject_unknown_client (use with caution)
• smtpd_helo_restrictions
• smtpd_helo_required yes
• smtpd_helo_restrictions
  reject_invalid_hostname
  reject_unknown_hostname
  reject_non_fqdn_hostname

49
Sample Rejected Logs
50
Postfix Spam Filtering Functions

• SMTP access restriction
• smtpd_sender_restrictions
• smtpd_sender_restrictions
  reject_unknown_sender_domain
  reject_non_fqdn_sender check_sender_access
  hash/etc/postfix/sender-access
• Useful to block SPAM mails that fake envelope
  mail from header

51
Postfix Spam Filtering Functions

• SMTP access restriction
• smtpd_recipient_restrictions
• smtpd_recipient_restrictions
  check_recipient_access regexp/usr/local/etc/postf
  ix/recp-access reject_unknown_recipient_domain
  reject_unauth_destination permit_sasl_authentic
  ated permit_mx_backup

52
Postfix Spam Filtering Functions

• Header check
• Both envelope and content headers
• main.cfheader_checks regexp/usr/local/etc/post
  fix/header

53
Postfix Spam Filtering Functions

• Simple content check
• main.cf
• body_checks regexp/usr/local/etc/postfix/body

• Content filtering interface
• http//www.postfix.org/FILTER_README.html
• Amavisd-new as an example

54
Virus and Spam Mail

• Virus mail
• Contain virus infected attachment
• How do we determine if the mail is virus
  infected?
• Simple and easy. Base on the inspection and
  judgment of antivirus software.
• Spam mail
• UCE/UBE
• Unsolicited Commercial Email
• Unsolicited Bulk Email
• How do we determine if the mail is a Spam mail?
• Difficult. Its not a yes/no problem.
• One persons meat may be anothers poison.

55
Issues When Dealing With SPAM

• Do not block or discard users mail unless you
  have the authorization and delegation.
• The only safe and acceptable assistant you can
  provide is to score and tag each mail and let
  your users to do the rest themselves.
• To warn or not to warn the senders/recipients is
  a dilemma!
• Virus and spam senders nowadays always fake the
  from address.
• System notification mails may be another kind of
  SPAM. The result will be blamed by your users and
  blocked by other mail servers.

56
Greylisting

• Recently proposed (2003) method to block spam and
  virus mails.
• RFC requires MTA to be capable to support queue
  and retry of temporarily undeliverable mails.
• Spam and virus always fire and forget.
• For efficiency concern, usually they do not
  retry.
• How it works?
• In brief, block every delivery at its first try.
• For each mail delivery, check if the the
  triplet(source ip, sender, recipient) was seen
  recently.
• If yes, pass the mail.
• If not, reject the delivery with 450 Please
  retry later.

57
Greylisting

• Drawback
• Mail delivery will suffer a delay ranging from
  several minutes to hours.
• Depending on the remote MTA configuration.
• Risk
• Sites that do not retry their mail delivery.
• Some bank notifications were observed to try only
  once.
• Mail sending programs that process the SMTP
  transaction themselves without calling the
  sendmail MTA interface.
• Suggestion
• Whitelisted known hosts to prevent from delay and
  loss of mail.

58
Greylisting

• Benefit
• Effectively reduce the volume of spam and virus.
• Effectively reduce both cpu and disk load of
  servers.

59
Softwares To Block/Filter Spam and Virus

• Postfix
• Content filter interface
• Postgrey
• Greylisting policy server
• Amavisd-new
• http//www.ijs.si/software/amavisd
• http//www.ijs.si/software/amavisd/README.postfix
• Clam Antivirus
• http//www.clamav.net
• A GPL virus scanner
• SpamAssassin
• http//spamassassin.apache.org

60
Software Architecture for Filtering
61
Amavisd-new

• A high performance interface between MTA and
  content checkers.
• Calling external antivirus programs to do virus
  scanning.
• Calling external spamassassin program to do spam
  level determination.
• CPU intensive workloads.
• Can be flexibly configured to pass, discard, or
  quarantine mails based on user defined policy.
• Pass spam mails with score gt 10 with subject
  prepended the SPAM keyword.
• Quarantine spam mails with score gt 20.
• Discard spam mails with score gt 30.
• Quarantine virus mails.

62
Spamassassin

• Spam level scoring software.
• Rich set of tests to identify various spam
  signatures.
• Keywords, bad headers, encodings
• Use bayesian analysis to help scoring.
• Training the bayesian database using know spam
  and ham mails.
• Default to enable the auto-learn feature.
• Calling external programs to check if the mail
  was a known spam.
• Use hash of mail content as the query key.
• Razor, DCC, Pyzor.

63
Spamassassin

• RBL(realtime black list) look up based on sender
  ip address.
• RBL may contains too many ill-administrated
  sites.
• Use the result as an addition of spam score.
• Do not block remote sites depend solely on RBL.
• SURBL(Spam URI realtime black list) look up based
  on the URIs within the content of mail.
• Spammers may keep changing their sending IP
  addresses.
• The URIs in the content may be the final
  destination the advertisement want people to
  visit.

64
Postfix Content Filter Configuration

• master.cf
• smtp-amavis unix - - y/n - 2 smtp -o
  smtp_data_done_timeout1200 -o
  smtp_send_xforward_commandyes -o
  disable_dns_lookupsyes
• 127.0.0.110025 inet n - y/n - - smtpd -o
  content_filter -o local_recipient_maps
  -o relay_recipient_maps -o
  smtpd_restriction_classes -o
  smtpd_client_restrictions -o
  smtpd_helo_restrictions -o
  smtpd_sender_restrictions -o
  smtpd_recipient_restrictionspermit_mynetworks,rej
  ect -o mynetworks127.0.0.0/8 -o
  strict_rfc821_envelopesyes -o
  smtpd_error_sleep_time0 -o
  smtpd_soft_error_limit1001 -o
  smtpd_hard_error_limit1000
• main.cf
• content_filter smtp-amavis127.0.0.110024

65
Amavisd-new Configuration

• amavisd.conf
• max_servers 30 number of pre-forked
  children
• _at_av_scanners ( .
• final_virus_destiny D_DISCARD (defaults
  to D_BOUNCE)
• final_banned_destiny D_BOUNCE (defaults to
  D_BOUNCE)
• final_spam_destiny D_DISCARD (defaults
  to D_REJECT)
• final_bad_header_destiny D_PASS (defaults
  to D_PASS), D_BOUNCE suggested
• QUARANTINEDIR '/var/virusmails/infected'
• sa_tag_level_deflt 1 add spam info
  headers if at, or above that level
• sa_tag2_level_deflt 9 add 'spam detected'
  headers at that level
• sa_kill_level_deflt 20 triggers spam
  evasive actions
• sa_dsn_cutoff_level 20 spam level beyond
  which a DSN is not sent
• sa_quarantine_cutoff_level 30
• Raise the tag2 value to avoid removing users
  mail.

66
SpamAssassin Configuration

• Built-in tests
• http//spamassassin.apache.org/tests.html
• local.cf
• ok_languages en ja zh
• ok_locales en ja zh
• score SUBJ_ILLEGAL_CHARS 0
• score FROM_ILLEGAL_CHARS 0
• score HEAD_ILLEGAL_CHARS 0
• score CHARSET_FARAWAY 1.0
• score CHARSET_FARAWAY_HEADER 1.0
• score MIME_CHARSET_FARAWAY 1.0
• header NTU_SMTP Received /mx3.cc.ntu.edu.tw
  \(relay5.tp1rc.edu.tw \163.28.16.251\\) \(using
  TLSv1 with cipher DHE-RSA-AES256-SHA \(256\/256
  bits\)\) . by mx1.cc.ntu.edu.tw \(Postfix\)/
• score NTU_SMTP -15.0
• describe NTU_SMTP mx3.cc.ntu.edu.tw
• header HINET_MSR Received /msr\d1,2.hinet.n
  et \(msr\d1,2.hinet.net \168.95.4.1\d1,2\\)
  by mx1.cc.ntu.edu.tw \(Postfix\)/
• score HINET_MSR -10.0
• describe HINET_MSR msr.hinet.net

67
Sample Log of Postfix

• Each incoming mail may have up to two transaction
  logs.

68
Sample Log of Amavisd
69
Sample Headers of Users Mail
70
Sample Headers of Users Mail
71
Sample Headers of Users Mail
72
Sample Screenshots of Outlook Express
73
Make Use of the Filter Rules on your MUA

• Outlook Express
• Can only filter based on limited headers.
• Subject, Sender, etc.
• Configure to move mails having subjects beginning
  with SPAM to an SPAM folder.
• Outlook, Netscape, Thunderbird
• You can filter mails based on any headers.
• Determine your own spam level threshold referring
  the X-Spam-Level

74
Conclusion

• Make use of public domain, open source software
  to construct your own mail system.
• Official web sites, mailing lists, and google are
  always your best friends.
• Feedback and contribute as possible as you can.