Using MSMQ as an SOA message bus

1
Using MSMQ as an SOA message bus

• Giora Tamir

2
About Giora Tamir

• Giora Tamir has been Architecting, Designing and
  Developing software and hardware solutions for
  over 15 years. As an IEEE Senior member and a
  talented developer, Giora blends software
  development, knowledge of protocols, extensive
  understanding of hardware and profound knowledge
  of both Unix and Windows based systems to provide
  a complete solution for both defense and
  commercial applications. Giora now holds the
  position of Principal Engineer for ProfitLine,
  Inc. architecting the next generation of .NET
  applications based on a Service-Oriented-Architect
  ure.
• Gioras areas of interest include distributed
  applications, networking and cryptography in
  addition to Unix internals and embedded
  programming.

3
About ProfitLine

• Founded in 1992, ProfitLine manages hundreds of
  millions of dollars in annual telecom spend for
  its prestigious Fortune 1000 client base, such as
  Ernst Young, Charming Shoppes, Macromedia, CNA
  Financial Corporation, and Constellation Energy
  Group.
• ProfitLine's outsourced solution streamlines
  telecom administrative functions by combining a
  best practices approach with intelligent
  technology.
• For more information about ProfitLine, call
  858.452.6800 or e-mail sales_at_profitline.com

4
Table of Content

• What is an SOA?
• SOA communication requirements
• What is MSMQ?
• What can MSMQ do out of the box?
• Using MSMQ as a message bus
• MSMQ best practices
• Troubleshooting MSMQ

5
What is a Service Oriented Architecture (SOA)?

• Business functionality is implemented using
  independent software modules (Services).
• Each independent software module is responsible
  for a single business function executed in the
  correct order.
• Implementation can be highly distributed.

6
SOA Comm. Requirements

• Flexible Architecture
• Support distributed services
• On multiple machines
• Multiple instances on one machine
• Messaging Functionality
• Support for both synchronous (request-reply) and
  asynchronous (fire-and-forget)
• One request must result in exactly one execution

7
SOA Comm. Requirements (2)

• Messaging Functionality (continued)
• Location independence
• Guaranteed order (or no order at all)
• Request expiration and timeouts
• Application level retry support
• Deadlock prevention and recovery
• Maintenance Goal
• Minimal setup and minimal maintenance when adding
  and removing services

8
Questions about SOA communications?
9
What is MSMQ?

• Microsoft Message Queuing (MSMQ) is a component
  of the Windows operating system that allows
  cooperating applications to send and receive
  messages to each other.
• Available in multiple flavors
• MSMQ 1.0 (Windows NT 4.0)
• MSMQ 2.0 (Windows 2000 Server)
• MSMQ 3.0 (Windows 2003 Server)

10
What is MSMQ? (2)

• Allows processes to communicate by sending and
  receiving messages
• Allows queues to be created to manage message
  ordering and delivery
• Supports guaranteed and non-guaranteed delivery
  of messages
• Understands transactions (poorly)
• Totally integrated with Active Directory

11
Built-In MSMQ functionality

• Very good .NET support (1.0, 1.1, 2.0)
• Messages contain multiple properties
• Body
• ID, Transaction ID, Correlation ID
• TimeToBeReceived (expiration)
• Time stamps
• Encryption
• Digital signatures, Authentication

12
(No Transcript)
13
Built-In MSMQ Functionality (2)

• Advanced send and receive
• Journaling for guaranteed delivery (Recoverable
  message support)
• Routing and multi-domain
• Reading, peeking and enumerating the content of
  queues
• Asynchronous operation (Begin and End)
• Multiple message formatting options (XML, Binary)

14
Built-In MSMQ Functionality (3)

• Advanced Queue Functionality
• Queues can be public (using Active Directory) or
  private (hidden).
• Queues can survive a restart (Messages are stored
  on-disk).
• Message expiration handled automatically.
• Multi-threading and multi-access safe
• Message access guarantee only a single process
  can remove a message from a queue.

15
Questions about MSMQ?
16
MSMQ as a Message Bus?

• The classic MSMQ messaging implementations
  assumes
• Application specific queues
• Polling for new messages on queues
• Correlation ID to handle replies
• Journaling to guarantee delivery.
• How can MSMQ be made into an SOA communications
  mechanism?

17
Make MSMQ an SOA Communications Mechanism

• Use one central queue (OK, maybe two)
• All messages are published to one place
• The Label property is used as the message
  subject
• A message intended for the accounting service
  handling new invoices might carry the label
  ACCOUNTING.NewInvoice.
• Services examine the queue looking for a label
  match
• The Accounting Service is looking for the
  ACCOUNTING.NewInvoice label

18
MSMQ Communications Mechanism (2)

• Use one central queue (continued)
• Multiple Accounting Service processes (possibly
  on multiple machines) may try to remove (read)
  the ACCOUNTING.NewInvoice message from the
  queue.
• Only one Accounting Service process will succeed
  in getting the message (go MSMQ!).
• One request results in (at most) one receive.

19
Make MSMQ an SOA Communications Mechanism (3)

• Services receive callback when messages are
  available on the queue
• Use BeginPeek and EndPeek to implement a callback
  trigger
• Call MessageQueue.BeginPeek() passing in your
  delegate (callback pointer)
• When called-back, call EndPeek(), examine the
  message and immediately call BeginPeek() again,
  passing in the same delegate as before
• No extra thread needed!

20
Make MSMQ an SOA Communications Mechanism (4)

• Message Expiration and Retries
• Set message expiration based on thedesired
  functionality
• Short expiration for interactive, time sensitive
  messaging.
• No expiration for fire-and-forget type messaging.
• Add application-level retry based on the business
  need
• Single attempt (for fire-and-forget)
• Multiple attempts (5, 10, every 30 seconds)

21
Make MSMQ an SOA Communications Mechanism (5)

• Examining messages on the Queue
• Look at the first (top) message
• Check all messages in the Queue
• If message order matters
• Examine only the first message (free if using
  EndPeek)
• If out-of-order reading is OK
• Enumerate the queue and examine all messages

22
Make MSMQ an SOA Communications Mechanism (6)

• Examining messages on the Queue (cont)
• Examining only the first message in the queue can
  lead to hanging and deadlocks
• If the first message is never picked up (and
  expiration is not set), all subsequent messages
  will never be delivered!
• If expiration is short and many messages are
  queued, messages can expire before a process can
  get to them (sequential processing).
• Additional (traditional) deadlocks possible.

23
Make MSMQ an SOA Communications Mechanism (7)

• Separate Queues
• One queue for interactive, short-lived messages
• Avoid having long-lived messages in the
  interactive queue.
• Limit the size of the interactive queue (40)
• Short queue ? fast enumeration
• No hangs due to a long-lived message stuck at
  the top of the queue
• A second queue for the long-lived messages

24
Make MSMQ an SOA Communications Mechanism (8)

• Application level reply messages
• The problem two separate receive mechanisms (an
  encapsulation problem)
• Receive by label (for normal messages)
• Receive by correlation ID (for replies)
• The solution always use receive-by-label
• Provide a reply label in the message body.
• Set the reply label to a GUID (unique).
• After sending the message, listen for a reply on
  the GUID label.

25
MSMQ and SOA - Recap

• A central queue (or two)
• Events (callbacks) instead of polling
• Message label allows publish-subscribe
• Many may examine, only one removes
• Appropriate expiration and retries
• Separate interactive and long lived
• Application level replies

26
MSMQ for SOA questions?
27
MSMQ Best Practices

• Read local, send remote
• Local reads are cheap, remote reads are expensive
• Remote sends are as cheap as local sends
• Switch to Windows 2003 and MSMQ 3.0
• Enumerating a remote queue using MSMQ 2.0, about
  500mSec per message
• Enumerating a remote queue using MSMQ 3.0, about
  10mSec per message
• A 10 message queue would cost 5 seconds in 2.0
  but only 0.1 seconds in 3.0!

28
MSMQ Best Practices (2)

• Use private queues
• Public Queues (computername\queuename)
• Stored in Active Directory
• Only a domain admin can create
• When attempting to access a public queue, the
  Active Directory (domain) controller is
  interrogated to find the queue
• Private Queues (computer\private\queue)
• Private queues are not stored in Active Dirctory
• Finding a private queue does not involve AD

29
MSMQ Best Practices (3)

• Use DIRECT queue paths
• computer\queue public queue (interrogates AD
  to find the queue)
• computer\private\queue private queue
  (interrogates AD to find the computer)
• FormatNameDIRECTOScomputer\private\queue
  private queue (do not use AD, use the operating
  system to find computer and connect directly to
  access the queue. Works even over the internet!
  NOTE does not support transactional queues.

30
Questions about Best Practices?
31
MSMQ Troubleshooting

• System.Messaging.MessageQueueException Queue is
  not registered in the DS.
• System.Exception Queue cannot be read!
• Queue not found most common error
• Check the path name any typos?
• Switch to DIRECT path names, Active Directory is
  known to occasionally go wild.
• System.Exception Queue cannot be read!
• Cannot access queue permissions
• By default, everyone is allowed to write to a
  queue but only the owner-creator can read from a
  queue.
• Allow everyone to read OR define a special user
  for MSMQ communications.

32
(No Transcript)
33
MSMQ Troubleshooting (2)

• Messages disappear (somewhere between the sender
  and the receiver)
• Use the Management Console to look at the queues
  and the messages
• Use a stand-alone message sending test program to
  send messages to the queue
• Use a stand-alone queue monitor to examine the
  messages in the queue
• Many cases caused by label mismatches or
  too-short expiration settings

34
(No Transcript)
35
(No Transcript)
36
MSMQ Troubleshooting (3)

• Message delivery stops everything is fine for a
  few hours (or even days) but at some point, new
  messages are no longer received.
• Is there a firewall involved?
• When using remote reads, the local MSMQ opens an
  RPC connection to the remote MSMQ. The connection
  is only active when new messages are added to
  the queue.
• If the queue is idle for a long period (say over
  the weekend), the intermediate firewall may
  decide to cut the connection.

37
Questions about Troubleshooting MSMQ?
38
The future of MSMQ

• MS-SQL Service Broker seems to be the
  next-generation MSMQ
• T-SQL commands to control queues and messages
• Proven SQL client libraries and protocols for
  communications
• Automatic triggering and spawning of services
• Guaranteed message ordering
• Improved transaction support

39
Where To Now?

• Tools code will be available on The Code
  Project (www.codeproject.com) in about two weeks.
• Search for author gtamir or look for MSMQ.
• Questions gioratamir_at_gmail.com
• MSMQ Resources
• http//msdn.microsoft.com System Messaging
• spaces.msn.com/members/msmq/