Managing Process Integrity (Chapter 8)

1
Managing Process Integrity(Chapter 8)

• B. Ramamurthy

2
Topics for discussion

• Review of last lectures
• Services layers diagram (Ch. 1-7)
• Webservices (design and deployment)
• Relate these two in the context of design,
  development, deployment cycle of an SOA.
• Lets understand WSDL
• http//www.w3.org/TR/wsdl_wsdl
• http//www.w3schools.com/wsdl/wsdl_documents.asp
• http//webservices.amazon.com/AWSECommerceService/
  AWSECommerceService.wsdl?
• Lets look at the components of WSDL and look at
  Amazon.com ECS as example
• Review for midterm
• Process Integrity

3
Anatomy of a WSDL document
4
Sample WSDL

• http//www.w3.org/2001/04/wsws-proceedings/uche/ws
  dl.html
• http//webservices.amazon.com/AWSECommerceService/
  AWSECommerceService.wsdl?

5
Significance of WSDL

• WSDL for a web service (WS) when specified in an
  IDE (such as Netbeans) exposes the operations and
  the signature for the operations.
• This can be used access the operations of the WS.
• When designing business systems using BPEL, WSDL
  represents the requirements of a process in a
  BPEL.
• Business ?BPEL ?WSLD ? services (SOA)
• Thus WSDL is the link that aligns a business
  process with an IT service.

6
Data integrity

• Consistency
• Accuracy
• Correctness of data
• Entity integrity each row in relational table
  must be uniquely identified
• Domain integrity the values must fall within a
  specified value range
• Referential integrity specifies the validity of
  relationship between different data sets.

7
Process integrity

• Business processes span multiple IT systems.
• There are times when a process state may be
  inconsistent. For example, a cancelled order
  takes some time to register with order system.
• Technical failures such as network failure
  system-wide solution
• Business exceptions must be handled at the
  process level WSDL fault provisions
• Special cases overbooking of a flight (cannot be
  handled by domain integrity.) Must be handled at
  code level.
• Process ownership is another problem. For
  example, consider a workflow application.

8
Process integrity solutions

• Logging and tracing
• ACID transactions
• Atomicity, consistency, isolation, durability
• Transaction monitors
• 2PC two-phase commit protocol is used to ensure
  ACID properties for transactions that span more
  than a single resource manager.
• Phase 1 all participating resource managers
  acquire locks on shared resources
• Depending on phase transaction coordinator
  informs participants whether to commit or
  rollback
• Phase 2 participants either commit or rollback
  to previous state depending on instruction from
  coordinator
• Issues performance, lack of support for
  long-lived transactions, integration of legacy
  systems, not suited for discontinuous network,
  use of multi-level and nested transactions

9
Process integrity solutions transactional steps

• Combination of transactional steps and persistent
  queue can improve reliability of complex
  processes.
• Transactional steps are a set of closely related
  activities executed in the context of a single
  complex transaction.
• Transactional steps facilitate the decomposition
  of complex, long-lived business processes into
  individual steps with short execution times and
  high transactional integrity.
• Transactional steps dramatically increase the
  robustness and flexibility of distributed
  systems.
• provide a great way of ensuring the integrity
  of individual process steps.

10
Process integrity solutions transaction chain
compensation

• ACID properties are too strong for complex
  transactions
• Chained transactions with compensating
  transactions offer a viable means of dealing with
  process integrity.
• Compensating transactions undo previously
  executed steps if a problem is encountered during
  the execution of a particular step in a
  transaction chain.
• http//www.serviceoriented.org/ws-transaction.html
  

11
Sagas

• Sagas are formal workflow models that build on
  the concept of chained transactions (steps).
• A Saga describes a workflow wherein each step
  associated with a compensating transaction.
• How do you compensate out of funds failure?

12
BPM and process integrity

• BPM (Business Process Management) platforms
  provides features that enable business analysts
  and developers to design, execute and manage
  instances of complex business process.
• Introduction of BPM will help the process
  integrity aspects of an enterprise application.
• Explicit modeling separates technical integrity
  from process integrity
• BPM engine provides mechanisms for monitoring
  processes at runtime
• BPM supports compensating transactions which is
  key concept for managing the rollback of
  partially executed processes after a failure.

13
Related WS-Standards

• See fig. 8-4
• A number of different industry alliances and
  standardization bodies are working on WS-based
  transaction protocols.
• OASIS Business Transaction Processing (BTP)
• WS Flow language
• WS-BPEL
• BPEL4WS
• XLANG
• WS-Composite Application Framework
• WS-coordination
• WS-Transaction

14
Process Integrity Spectrum
2PC/TPM in Heterogeneous env.
2PC/TPM in homogeneous env.
Level of integrity
Sagas with complex compensation
BPM engine with Simple compensation
Transactional Steps
Custom-built log infrastructure
DB log analyzer
Eyes closed Praying
Implementation cost
15
Mid-term review

• Date 11/29
• Duration 1 hour 30min
• Topics 1-7 Chapters
• Closed book
• Topics for questions
• Fundamental concepts synchrony, loose coupling
  etc., interface vs. payload semantics
• Service types
• Design of an SOA for a application layered
  architecture

16
Process Integrity in SOA Case Study Travel
Itinerary Management

1. Analyze the problem
2. Choose concurrency control optimistic vs
   pessimistic
3. Use idempotent update operations
4. Avoid distributed 2PC
5. Build transactional steps
6. Design simple and flexible compensations
7. Combine SOA, MOA and BPM for flexibility

17
Problem Statement Analysis (8.3.1 Fig.8-5)

• Typical sites enable you to book individual
  flights.
• The new travel itinerary mgt. system should
  create complete itineraries for trips including
  car, hotel and flights.
• Reuse existing IT subsystems esp. customer
  management and billing systems.
• Two different front-ends simple html front and a
  complex front-ends for call center (VB GUI)
  functionality
• Three backend systems customer management,
  billing, and other complex processes such as
  customer complaints etc.)
• A number of partner systems need to be
  integrated example partner airlines, hotel and
  car reservation systems.
• Key transactions are confirm itinerary
  (typically irreversible) and create invoice
  these transactions
• cross organizational boundaries
• cross several technical boundaries
• involve different databases
• involve different departments

18
Architectural model
Enterprise Reservation systems (cars, hotels,
other airlines)
Enterprise Reservation systems (cars, hotels,
other airlines)

Customer
Customer Itinerary DB
Web front-end

Billing
Callcenter front-end
Invoice DB
19
Concurrency Control (8.3.2, figs.8-6, 8-7)

• This specifies how to deal with concurrent access
  to shared data in an SOA.
• Optimistic and pessimistic
• Pessimistic control gives exclusive access to
  data through acquisition of locks locks
  typically are held for short duration.
• Optimistic control no locks are acquired during
  the transaction execution, only when data needs
  to be updated at the end of the transaction.
• Optimistic control is the choice for long running
  transactions.

20
Implementing Optimistic Concurrency Control

• In order to determine write conflicts, optimistic
  concurrency model must, at the end of each
  transaction, determine whether a competing
  transaction has changed the data after it was
  initially checked out.
• Different ways timestamps, version counts, state
  comparisons
• For the first two an extra column is added in the
  relational table
• Version number or timestamp is added to the
  primary key used in the WHERE clause of UPDATE
  operation
• State comparison is implemented in Microsofts
  ADO.NET advantage does not require extra column
  in the relational table
• Version numbers are associated with a service at
  root level
• If a client is a trusted party then version
  number is preferable solution else state
  comparison

21
Example

• Customer profile
• Customer updates using web call center updates
  using telephone
• Customer and call center read profile version
  645
• Customer updates, saves, version 646
• Call center looses its updates, needs redo due
  version conflict
• Alternatively, we could allow merge operation
  possible with state comparison
• If nothing works resort to pessimistic control,
  such as in an incident management service

22
Optimistic concurrency control
Merge possible
Notify user
yes
yes
merge changes
23
Pessimistic concurrency control
Get a lock on Incident object
Lock Avail?
no
Notify user
yes
yes
Work on Incident
Possibility for discard changes as well as
supervisor notification
no
Commit changes Release lock
24
Make updates Idempotent

• Effect of a N (N gt 1) of the same requests is
  same as one request.
• Use sequence numbers
• interface Itinerary
• SQN getSeqeunceNumber()
• void addBooking (in SQN s, in booking b)
• Can be used in error handling

25
Idempotent Operation in error handling
while (retry limit not reached) try //
two idempotent operations in a block
itineraryManager.confirmItinerary()
invoiceManager.createInvoice() catch
(FatalError e) //manage retry limit
counter if (not successful) // handle
error scenarios
26
Avoid Distributed 2PC

• First iteration Client controlled transactions
  (fig. 8.9)
• Has a transactions manager which implements the
  2PC begin commit commit
• Client should deal with distributed transactions
• Lacks loose coupling
• Second iteration server controlled transactions
  (fig. 8.10)
• Transactions spanning multiple databases combined
  on the server side
• Tightly coupled, still 2PC, cost of sophisticated
  transaction monitor

27
Building Transactional Steps

• Split transactions and introduce queues
• confirm_itinerary() operation stores a message in
  a pending confirmations queue
• A background thread uses this queue as input and
  executes operation and stores in confirmed
  itineraries queue
• System picks from this queue and informs customer
  of success or failure
• Problem a transaction monitor is needed for 2PC
  among queues and DB
• Solution introduce services at various layers

28
Granularity Level
BEING pendingConfirmQ.get() try
itineraryMgr.confirm_itinerary() catch
(e1) errorQ.put(e1) return COMMIT try
invoiceMgr.create_invoice(r1)
catch (e2) errorQ.put(e2) return
COMMIT confirmedItineraryQ.put() COMMIT
29
Compensating Logic

• Use simple compensating logic
• Confirm itinerary ? Cancel itinerary
• Create invoice ?cancel invoice
• Compensating logic may not work all the time. In
  such cases treat them as special cases at the
  application level (see 8-16)

30
SOAMOABPM

• Combine SOA (services) , MOA (messages and
  queues) and BPM (business processes, compensation
  and special handling) to increase flexibility
• Read Ch.8 and review the figures explaining these
  concepts