Chapter 26: SMTP and a Brief Introduction to POP, IMAP and MIME

1
Chapter 26 SMTP and a Brief Introduction to
POP, IMAP and MIME

• Clemson Lugtu
• Jeff Hahn
• Jeremy Uziel

2
What is Electronic Mail?

• Electronic Mail - It is a facility that allows
  users to
• transmit messages across the Internet.
• It is the most widely used application service
  offering
• a fast conventional method of transferring
  information.
• It can accommodate large messages or voluminous
  memos.
• It allows communication between groups or single
• individuals.

3
Spooling

• Electronic mail systems use a technique known as
• SPOOLING to handle delayed delivery
• When user sends a mail, the system places a copy
  in
• its private storage (also known as a spool)
• The system initiates transfer to remote machine
  as
• a background activity.
• The background mail transfer process becomes
  client.
• If succeeded, the transfer process passes a copy
  of
• the message to the remote server.

4
Spooling

• If it fails, the transfer process records the
  time
• delivery was attempted and terminates.
• The background transfer process sweeps through
  the
• spool area periodically. (usually 30
  minutes)
• A spool also contains the identification of the
  sender,
• recipient, destination machine, and the time
  a mail
• message was deposited.
• When it finds a message or when a user deposits
  new
• outgoing mail, it attempts delivery.
• If a mail message can't be delivered after an
  extended
• time, the mail software returns the mail
  message to
• the sender.

5

6
Electronic Mail Addresses

• A receiver is identified by an email address such
  as
• local-part _at_ domain-name
• domain-name Specifies a mail exchanger. It
  determines
• the domain name of a mail destination to
  which mail
• should be delivered.
• local part Specifies a mailbox on the mail
  exchanger.
• It is often identical to a login name or a
  full name
• of a user.

7

• Example electronic mail addresses
• clem_at_csun.edu
• comer_at_purdue.edu
• john_at_example.com
• bill_at_college.edu
• clem, comer, and john distinguish the local part
  of an
• electronic mail address.
• csun.edu, purdue.edu, and example.com
  distinguish the
• domain-name part of an email address.

8
Alias expansion and mail forwarding

• Most systems provide (mail forwarding) software
  that
• includes a (mail alias expansion) mechanism.
• A (mail forwarder) allows a local site to map
  identifiers
• used in mail addresses to a set of one or
  more new
• mail addresses.
• After a user composes a message and names a
  recipient,
• the (mail interface program) talks to the
  local aliases
• to replace the recipient with the mapped
  version before
• passing the message to the delivery system.
• Recipients to which no mapping was provided stays
  
• unchanged.

9
TCP/IP Standards forElectronic Mail Service

• TCP/IP provides interoperability for electronic
  mail.
• TCP/IP divides its mail standards into two sets.
  One standard specifies the format for mail
  messages. The other standard specifies the
  details of electronic mail exchange between two
  computers
• Each memo is divided into two parts header and
  body
• separated by a blank line
• header must contain "From" and "To" fields.
• body The format of the body is left to the
  sender

10
Post Office ProtocolVersion 3 (POP3)

• Post Office Protocol Version 3 has been the most
  popular protocol has been the most popular
  protocol used to transfer messages from a
  permanent mailbox to a local computer.
• When the user invokes a POP3 client, it
  establishes a TCP connection and contacts a POP3
  server on the mailbox computer. The user sends a
  login and a password to authenticate the session.
  If login was successful, the client sends
  commands to retrieve copies of messages and to
  delete messages from the permanent mailbox.
• Both SMTP servers and POP3 servers must
  synchronize access to the mailbox.

11
POP3 Commands

• USER name User name for authentication
• PASS password Password used for authentication
• STAT Get number and total size of message
• LIST msg get size of message
• RETR msg Send message to client
• DELE msg Delete message from mailbox
• RSET Cancel previous delete requests.
• QUIT Updates mailbox (deletes messages) and
  quits.

12
Internet Message AccessProtocol (IMAP)

• An alternative to POP3 is IMAP version 4. It
  defines an
• abstraction known as a MAILBOX. Mailboxes
  are located on
• the same computer as a server.
• IMAP4 is a method for accessing electronic mail
  messages
• that are kept on a mail server. It permits a
  client
• e-mail program to view and manipulate those
  messages.
• Electronic mail stored on an IMAP server can be
  viewed
• or manipulated from a desktop computer at
  home,
• a notebook computer, or at a workstation. We
  can also say
• that mail messages can be accessed from
  multiple
• locations.

13
What are the functionsof IMAP4?

• Includes operations for
• creating mailboxes
• deleting mailboxes
• renaming mailboxes
• checking for new messages
• permanently removing messages
• setting and clearing flags
• searching
• fetching of message attributes texts, and
  portions.

14

• IMAP provides extended functionality for message
  retrieval and processing.
• Users can obtain information about a message or
  examine header fields without retrieving the
  entire message.
• Users can search for a specified string and
  retrieve portions of a message. This is useful
  for slow-speed dialup connections since they wont
  need to download useless information.

15
Multipurpose Internet Mail Extensions (MIME)

• MIME is defined to allow transmission of
  non-ASCII or arbitrary data through a standard
  e-mail message.
• In other words, MIME has a mechanism for sending
  multimedia data over e-mail.

16
MIME Messages

• MIME information is contained in the mail header
  using standard RFC 2822 format.
• MIME header specifies version, data type,
  encoding used to convert the data to ASCII.

17
Header Lines ofMIME Messages

• MIME-Version shows how message was composed
  using the
• version 1.0 of the MIME protocol.
• Content-Type Specifies the 7 basic content types
  of data in
• the message
• Text (a simple text document)
• image (photograph or computer generated image)
• audio (sound recordings)
• video (video recordings)
• application (raw program data)
• multipart (multiple messages with each having a
  seperate
• content type and recording.
• message (an entire e-mail message)
• Content-Type must have 2 identifiers, a
  content-type
• and a subtype. type/subtype(i.e.
  image/jpeg, audio/wav,
• image/gif)

18
Header Lines ofMIME Messages

• Content-Transfer-Encoding type of encoding that
  was used
• (base64 for JPEG images) to convert the data to
  ASCII.
• Example
• From bill_at_college.edu
• To john_at_example.com
• MIME-Version 1.0
• Content-Type image/jpeg
• Content-Transfer-Encoding base64
• ..data for the image..

19
MIME Multipart Messages

• MIME multipart messages within the Content-Type
  adds
• considerable flexibility.
• There are four subtypes for a multipart message.
  The four
• subtypes are
• mixed allows a single message to contain
  multiple,
• independent sub-messages each having its
  independent type
• and encoding.
• alternative allows a single message include
  multiple
• representations of the same data.

20
MIME Multipart Messages

• parallel allows a single message to include
  subparts that should be viewed together. (such as
  video and audio subparts)
• digest allows a single message to contain a set
  of other messages.
• To summarize, a multipart message can contain
  both a short text explaining the purpose of the
  message and some non-textual information

21

• From carrieerpdgns_at_yahoo.com
• To john_at_example.com
• MIME-Version 1.0
• Content-Type Multipart/Mixed
  BoundaryStartOfNextPart
• --StartOfNextPart
• Content-Type text/plain
• Content-Transfer-Encoding 7bit
• John,
• Here is the photo of the carrier pigeons I saw
  last
• week.
• Sincerely,
• Carrie Erpigeons
• --StartOfNextPart
• Content-Type image/gif
• Content-Transfer-Encoding base64

22
E-mail attachments

• An e-mail client can allow you to add attachments
  to
• e-mails you send and to save attachments from
  e-mails
• you receive.
• An attachment usually is not text. (Otherwise, it
  would
• be added in the body of the message) Attachments
• can be referred to as a word document, sound
  files,
• images, or pieces of software.
• Since e-mail messages can only contain textual
  information,
• it can produces a problem.

23
E-mail attachments

• Solutions
• Solve problem by using a program called uuencode.
  
• It assumes that the file contains binary
  information.
• It extracts three bytes from the binary file and
  converts it to four text characters.
• uuencode produces an encoded version of the
  original binary file containing only text
  characters.
• recipient would save the uuencoded version of the
  message to a file and run uudecode to translate
  it back to binary.
• Since uuencoding and uudecoding messages have
  been done in the early days of e-mail, most
  modern e-mail clients run uuencode and uudecode
  for you automatically.

24
Introduction to SMTP

• The TCP/IP Protocol specifies a standard for the
  exchange of mail between machines. This standard
  specifies the exact format of messages a client
  on a single machine uses to transfer mail to a
  server on another.
• This standard transfer protocol is known as the
  Simple Mail Transfer Protocol. (SMTP) The main
  objective of SMTP is to provide the reliability
  and efficiency of mail transfer.

25
Introduction to SMTP

• What should or what shouldnt SMTP specify?
• SMTP basically focuses on how the underlying mail
  delivery
• system passes messages across an internet from
  machine to
• machine.
• SMTP does not specify how the mail system accepts
  mail
• from a user or how the user interface presents
  messages.
• It also does not specify how mail is stored or
  how frequent
• a mail system accepts to send messages.

26
Introduction to SMTP

• An important feature of SMTP is "mail relaying."
• Mail relaying is SMTP's capability of
  transporting across
• networks.
• Through SMTP, a process can transfer mail to
  another
• process using the same or other networks via a
  relay
• process accessible to both networks.
• Mail messages can be passed through a number of
  intermediate
• relay or gateways hosts from sender to ultimate
  destination.

27
Basic SMTP Design
User
Client SMTP
Server SMTP
SMTP Commands/Replies
and Mail
File System
File System
28
Relaying

• A relay SMTP server is usually the target of a
  DNS MX record that designates it, rather than the
  final delivery system. The relay server may
  accept or reject the task of relaying the mail in
  the same way it accepts or rejects mail for a
  local user.
• If it accepts the task, it then becomes an SMTP
  client, establishes a transmission channel to the
  next SMTP server specified in the DNS, and sends
  it the mail.
• If it declines to relay mail to a particular
  address for policy reasons, a 550 response should
  be returned.

29
Relaying

• If an SMTP server has accepted the task of
  relaying the mail and later finds that the
  destination is incorrect or that the mail cannot
  be delivered for some other reason, then it MUST
  construct an "undeliverable mail" notification
  message and send it to the originator of the
  undeliverable mail (as indicated by the
  reverse-path).

30
Mail Gatewaying

• A "gateway" SMTP system (usually referred to just
  as a "gateway") receives mail from a client
  system in one transport environment and transmits
  it to a server system in another transport
  environment.
• Differences in protocols or message semantics
  between the transport environments on either side
  of a gateway may require that the gateway system
  perform transformations to the message that are
  not permitted to SMTP relay systems. Firewalls
  that rewrite addresses should be considered as
  gateways, even if SMTP is used on both sides of
  them.

31
Address Verification

• In some hosts the distinction between a mailing
  list and an alias for a single mailbox is a bit
  fuzzy, since a common data structure may hold
  both types of entries, and it is possible to have
  mailing lists containing only one mailbox.
• SMTP provides commands to verify a user name or
  obtain the content of a mailing list.
• The VRFY and EXPN commands, which have character
  string arguments, are commands that can be used
  to verify mailboxes or mailing lists.

32
Address Verification

• When normal responses are returned from a VRFY or
  EXPN request, the reply normally includes the
  mailbox name, i.e., "ltlocal-part_at_domaingt or
  User Name ltlocal-part_at_domaingt, where "domain"
  is a fully qualified domain name

33
VRFY Command

• VRFY Responses from Server
• 553 User ambiguous
• or
• 553- Ambiguous Possibilities are
• 553-Joe Smith ltjsmith_at_foo.comgt
• 553-Harry Smith lthsmith_at_foo.comgt
• 553 Melvin Smith ltdweep_at_foo.comgt

34
EXPN Example

• C EXPN Example-People
• S 250-Jon Postel ltPostel_at_isi.edugt
• S 250-Fred Fonebone ltFonebone_at_physics.foo-u
  .edugt
• S 250 Sam Q. Smith ltSQSmith_at_specific.generi
  c.comgt
• or
• C EXPN Executive-Washroom-List
• S 550 Access Denied to You.

35
EHLO Command

• A client SMTP SHOULD start an SMTP session by
  issuing the EHLO command.
• If the SMTP server supports the SMTP service
  extensions it will give a successful response, a
  failure response, or an error response.
• If the SMTP server, in violation of this
  specification, does not support any SMTP service
  extensions it will generate an error response.

36
HELO Command

• Older client SMTP systems may use HELO (as
  specified in RFC 821) instead of EHLO, and
  servers MUST support the HELO command and reply
  properly to it.
• In any event, a client MUST issue HELO or EHLO
  before starting a mail transaction.

37
Mail Command

• This command is used to initiate a mail
  transaction in which the mail data is delivered
  to an SMTP server which may, in turn, deliver it
  to one or more mailboxes or pass it on to another
  system (possibly using SMTP).
• The argument field contains a reverse-path and
  may contain optional parameters.
• The reverse-path consists of the sender mailbox.

38
RCPT Command

• The RCPT (Recipient) command is used to identify
  an individual recipient of the mail data
  multiple recipients are specified by multiple use
  of this command.
• The argument field contains a forward-path and
  may contain optional parameters.

39
DATA Command

• The receiver normally sends a 354 response to
  DATA, and then treats the lines following the
  command as mail data from the sender.
• The mail data may contain any of the 128 ASCII
  character codes, although experience has
  indicated that use of control characters other
  than SP, HT, CR, and LF may cause problems and
  SHOULD be avoided when possible.

40
DATA Command

• The mail data is terminated by a line containing
  only a period, that is, the character sequence
  "ltCRLFgt.ltCRLFgt" or "ltCRgtltLFgt.ltCRgtltLFgt"

41
NOOP Command

• This command does not affect any parameters or
  previously entered commands.
• It specifies no action other than that the
  receiver send an OK reply.

42
RSET Command

• The RSET (Reset) command specifies that the
  current mail transaction will be aborted.
• Any stored sender, recipients, and mail data MUST
  be discarded, and all buffers and state tables
  cleared.
• The receiver MUST send a "250 OK" reply to a RSET
  command with no arguments. A reset command may
  be issued by the client at any time.

43
QUIT Command

• This command specifies that the receiver MUST
  send an OK reply, and then close the transmission
  channel.
• The QUIT command may be issued at any time.

44
SMTP Example
45
Terminating Sessions and Connections

• An SMTP connection is terminated when the client
  sends a QUIT command. The server responds with a
  positive reply code, after which it closes the
  connection.
• If a server receives unknown commands it should
  not terminate connection, but it should issue a
  500 reply and await further instructions from the
  client.

46
Terminating Sessions and Connections

• An SMTP server MUST NOT intentionally close the
  connection except
• - After receiving a QUIT command and
  responding with a 221 reply.
• - After detecting the need to shut down the
  SMTP service and returning a 421 response code.
  This response code can be issued after the
  server receives any command or, if necessary,
  asynchronously from command receipt (on the
  assumption that the client will receive it after
  the next command is issued).

47
SMTP Replies

• Replies to SMTP commands serve to ensure the
  synchronization of requests and actions in the
  process of mail transfer and to guarantee that
  the SMTP client always knows the state of the
  SMTP server.
• Every command MUST generate exactly one reply.

48
Size Limits and Minimums

• Local part 64 characters
• Domain 255 characters
• Path 256 characters
• Command line 512 characters
• Text line 1000 characters
• Message content 64K octets

49
Errors reported byreply codes.

• Errors reported by reply codes.
• Examples
• 500 Line too long.
• 501 Path too long.
• 452 Too many recipients.
• 552 Too much mail data.
• In RFC 821 there were errors in the reply codes

50
Timeouts

• Initial 220 Message 5 minutes.
• Mail Command 5 minutes.
• RCPT Command 5 minutes.
• DATA Initiation 2 minutes.
• Data Block 3 minutes.
• DATA Termination 10 minutes.

51
Retry strategies

• Multiple strategies possible depending on your
  needs and system.
• Any queuing strategy must include timeouts on all
  activities on a per-command basis. A queuing
  strategy must not send error messages in response
  to error messages under any circumstances.
• Shouldn't constantly retry every entry in the
  queue at every cycle or it would cause a giant
  overhead.

52
Sending strategies

• In a typical system, the program immediately
  tries to send the message, but if it can't, it
  will enter it in a queue.
• Queue entry contains the message and the envelope
  information.
• The queues must be large.
• Typical resend time is 30 minutes but may be
  changed based on system.
• Typical max time for a message to be in the queue
  is 4-5 days but is configurable.
• A client should keep a list of hosts it cannot
  reach and corresponding connection timeouts,
  rather than just retrying queued mail items.
• When sending to multiple recipients, instead of
  sending mail, recipient, data, mail, recipient,
  data... it should send it as mail, recipient,
  recipient... recipient, data.
• Concurrent mailing is possible but should be
  limited so it doesn't tie up system resources.

53
Receiving strategies

• Should attempt to keep a pending listen on its
  port at all times.
• Requires support for multiple incoming TCP
  connections.
• When the SMTP server receives mail from a
  particular host address, it could activate its
  own SMTP queuing mechanisms to retry any mail
  pending for that host address

54
Messages with a nullreverse path

• Several notification methods required.
• Non-Delivery Notifications
• Delivery Status Notification
• Message Disposition Notification

55
Address resolution and mail handling

• Once an SMTP client lexically identifies a domain
  to which mail will be delivered for processing, a
  DNS lookup must be performed to resolve the
  domain name.
• The names are expected to be fully-qualified
  domain name.
• When the lookup succeeds, the mapping can result
  in a list of alternative delivery addresses
  rather than a single address, because of multiple
  MX records, multihoming, or both.
• To provide reliable mail transmission, the SMTP
  client must be able to try (and retry) each of
  the relevant addresses in this list in order,
  until a delivery attempt succeeds.
• There may also be a configurable limit on the
  number of alternate addresses that can be tried.
  In any case, the SMTP client should try at least
  two addresses.

56
Problem Detection and Handling
57
Reliable delivery and repliesby e-mail

• When the receiver-SMTP accepts a piece of mail,
  it is accepting responsibility for delivering or
  relaying the message.
• It must not lose the message for frivolous
  reasons, such as because the host later crashes
  or because of a predictable resource shortage.
• If there is a delivery failure after acceptance
  of a message, the receiver-SMTP must formulate
  and mail a notification message.
• This notification must be sent using a null
  ("ltgt") reverse path in the envelope.
• The recipient of this notification must be the
  address from the envelope return path (or the
  Return-Path line).
• If this address is null ("ltgt"), the receiver-SMTP
  must not send a notification.

58
Reliable delivery and repliesby e-mail

• If the address is an explicit source route, it
  must be stripped down to its final hop.
• Example
• Suppose that an error notification must be sent
  for a message that arrived with
• MAIL FROMlt_at_a,_at_buser_at_dgt
• The notification message must be sent using
• RCPT TOltuser_at_dgt
• Some delivery failures after the message is
  accepted by SMTP will be unavoidable.
• To avoid receiving duplicate messages as the
  result of timeouts, a receiver-SMTP must seek to
  minimize the time required to respond to the
  final ltCRLFgt.ltCRLFgt end of data indicator.

59
Loop detection

• Simple counting of the number of "Received"
  headers in a message has proven to be an
  effective, although rarely optimal, method of
  detecting loops in mail systems. SMTP servers
  using this technique should use a large rejection
  threshold, normally at least 100 Received
  entries. Whatever mechanisms are used, servers
  must contain provisions for detecting and
  stopping trivial loops.

60
Compensating forirregularities

• Variations, creative interpretations, and
  violations of Internet mail protocols do occur .
• There is a debate whether SMTP should drop the
  message, resend the message as is or repair and
  send.
• SMTP will repair messages that are incorrect or
  incomplete.
• The following changes to a message being
  processed may be applied when necessary by an
  originating SMTP server, or one used as the
  target of SMTP as an initial posting protocol
• Addition of a message-id field when none appears
• Addition of a date, time or time zone when none
  appears
• Correction of addresses to proper FQDN format
• The less information the server has about the
  client, the less likely these changes are to be
  correct and the more caution and conservatism
  should be applied when considering whether or not
  to perform fixes and how.
• These changes must not be applied by an SMTP
  server that provides an intermediate relay
  function.

61
Security Considerations
62
(No Transcript)
63
Mail security and spoofing

• SMTP is not a secure protocol.
• A user may spoof, which means they may trick the
  protocol to send a message posing as someone
  else.
• Usually a user will prove that they are authentic
  in the message body with a digital signature or
  SPF (sender policy framework) records.
• SPF records work by telling people which machines
  you send email from, and if the sender is not one
  of those machines then they are lying.
• If relay is on, no authentication possible.
• Spam was a huge problem early on because early
  defaults had relay on and people didn't know how
  to turn it off.

64
VRFY, EXPN and security

• For security reasons, a server might want to
  disable VRFY and EXPN
• Implementations of the above must not appear to
  have verified addresses that are not, in fact,
  verified.
• The SMTP server must return a 252 response,
  rather than a code that could be confused with
  successful or unsuccessful verification.
• Returning a 250 reply code with the address
  listed in the VRFY command after having checked
  it only for syntax violates this rule.
• An implementation that "supports" VRFY by always
  returning 550 whether or not the address is valid
  is equally not in conformance.
• Within the last few years, the contents of
  mailing lists have become popular as an address
  information source for "spammers."
• The use of EXPN to "harvest" addresses has
  increased as list administrators have installed
  protections against inappropriate uses of the
  lists themselves.
• Implementations should still provide support for
  EXPN, but sites should carefully evaluate the
  tradeoffs.
• As authentication mechanisms are introduced into
  SMTP, some sites may choose to make EXPN
  available only to authenticated requestors.

65
Information disclosure in announcements

• There has been an ongoing debate about the
  tradeoffs between the debugging advantages of
  announcing server type and version in the
  greeting response or in response to the HELP
  command and the disadvantages of exposing
  information that might be useful in a potential
  hostile attack.
• Those who argue for making it available point out
  that it is far better to actually secure an SMTP
  server rather than hope that trying to conceal
  known vulnerabilities by hiding the server's
  precise identity will provide more protection.
• Sites are encouraged to evaluate the tradeoff
  with that issue in mind
• Implementations are strongly encouraged to
  minimally provide for making type and version
  information available in some way to other
  network hosts.

66
Information disclosure intrace fields

• In some circumstances, such as when mail
  originates from within a LAN whose hosts are not
  directly on the public Internet, trace
  ("Received") fields produced in conformance with
  this specification may disclose host names and
  similar information that would not normally be
  available.
• This ordinarily does not pose a problem, but
  sites with special concerns about name disclosure
  should be aware of it.
• The optional FOR clause should be supplied with
  caution or not at all when multiple recipients
  are involved lest it inadvertently disclose the
  identities of "blind copy" recipients to others.

67
Information disclosure inmessage forwarding

• Use of the 251 or 551 reply codes to identify the
  replacement address associated with a mailbox may
  inadvertently disclose sensitive information.
• Sites that are concerned about those issues
  should ensure that they select and configure
  servers appropriately.

68
Scope of operation of SMTP servers

• An SMTP server may refuse to accept mail for any
  operational or technical reason that makes sense
  to the site providing the server.
• If sites take excessive advantage of the right to
  reject traffic, the ubiquity of email
  availability (one of the strengths of the
  Internet) will be threatened.
• Considerable care should be taken and balance
  maintained if a site decides to be selective
  about the traffic it will accept and process.
• In recent years, use of the relay function
  through arbitrary sites has been used as part of
  hostile efforts to hide the actual origins of
  mail.
• Some sites have decided to limit the use of the
  relay function to known or identifiable sources,
  and implementations should provide the capability
  to perform this type of filtering.
• When mail is rejected for these or other policy
  reasons, a 550 code should be used in response to
  EHLO, MAIL, or RCPT as appropriate.

69
SMTP vs. Carrier Pigeons