Fault Tolerant CORBA

1
Fault Tolerant CORBA

• Response to the
  Request for Proposals

Presentation to Real Time SIG realtime/99-11-02
11/14/1999 see also orbos/99-11-04
2
Basic Fault Tolerance Concepts

• Fault Tolerance Domains
• Object Groups
• Fault Tolerance Properties
• Strong Replica Consistency

3
Fault Tolerance Domains

• Aid application management and provide for
  scalability
• Each Fault Tolerance Domain is managed by a
  single ReplicationManager

Hawaii Location
Host 3
E2
F1
A1
B2
C1
Host 7
Host 2
Host 1
Host 5
IIOP Message over TCP/IP
Gate way
B1
Host 6
E1
C3
ORB without support for Fault Tolerance
C2
D1
E3
F2
Host 4
Los Angeles Domain
Wide Area Domain
Boston Domain
4
Object Groups

• Replicas of an object form an object group
• Each object group has an Interoperable
  Object Group Reference (IOGR)
• Object group abstraction provides
• Replication transparency
• Failure transparency

5
Fault Tolerance Properties

• Each object group has an associated set of fault
  tolerance properties, including
• Replication Style
• Membership Style
• Consistency Style
• Fault Monitoring Style
• Fault Monitoring Granularity
• Factories
• Initial Number of Replicas
• Minimum Number of Replicas
• Fault Monitoring Interval
• Checkpoint Interval

6
Strong Replica Consistency

• Maintained for object groups that have the
  Infrastructure-controlled Consistency Style
• For Passive replication, at the end of each state
  transfer, all of the members of the object group
  have the same state
• For Active replication, at the end of each
  operation, all of the members of the object
  group have the same state

7
Architectural Overview
create_ object()
set_ properties()
Replication Manager
Fault Notifier
Fault Detector
notifications
create_ object()
is_alive()
fault reports
Client
Server
Server
C
S1
S2
Factory
Fault Detector
Factory
Fault Detector
CORBA ORB
CORBA ORB
CORBA ORB
Logging Mechanism
Recovery Mechanism
Logging Mechanism
Recovery Mechanism
Logging Mechanism
8
1. Does the proposal extend, or respecify,
adopted OMG specifications?
Mandatory Requirements

• The proposal extends the existing OMG
  specification of CORBA with
• Three new IOR Components
• Two new Service Contexts
• IIOP Failover under Fault Conditions

9
2. What is the unit of redundancy?

• The object
• Objects are replicated
• The replicas of an object form an object
  group

10
3. What interfaces are provided to allow the
constitution of entity groups?

• The GenericFactory interface of the
  ReplicationManager allows the creation of
• Object groups
• Individual members of an object group
• The ObjectGroupManager interface allows
• The creation of object group members
  at specific locations
• The addition of objects as members of object
  groups

11
Infrastructure-Controlled Membership Style
Application directs ReplicationManager to create
an object group ReplicationManager creates and
adds the members to the group
Profile
Profile
IOGR
create_object()
Host Q
Host P
create_object()
Factory
Factory
12
Application-Controlled Membership Style
Application directs the
ReplicationManager to create a member
at a specific location and add
it to the group
Profile
Profile
IOGR
create_member()
Host Q
Host P
create_object()
Factory
13
Application-Controlled Membership Style
Application creates a member and directs
the ReplicationManager to add it to the group
Profile
Profile
IOGR
add_member()
Host Q
Host P
Factory
create_object()
14
4. What interfaces assist with state
synchronization within an entity group during
normal operation, and under failure and
recovery conditions?

• For an object group that has the
  infrastructure-controlled ConsistencyStyle,
  during normal operation, the Fault Tolerance
  Infrastructure maintains state synchronization
• During failure and recovery conditions, the
  Checkpointable and Updateable interfaces,
  which are inherited by the application objects,
  assist with state synchronization

15

• For an object group that has the
  application-controlled ConsistencyStyle, it is
  the responsibility of the application to provide
  whatever form of state synchronization is
  appropriate for the application both during
  normal operation and under failure and recovery
  conditions

16
5. What interfaces are provided for the
admittance of a new member, ensuring that the
states of all entities are equivalent?

• For the infrastructure-controlled
  MembershipStyle, the ReplicationManager
  automatically adds the new member to the object
  group to maintain the MinimumNumberReplicas
• For the infrastructure-controlled
  ConsistencyStyle, the Recovery Mechanism ensures
  that the state of the new member is the same as
  that of the existing members, using the
  Checkpointable and Updateable interfaces

17
6. What interfaces allow entity redundancy to
be transparent to clients?

• The object group abstraction hides object
  replication, faults and recovery from clients
• Many interfaces
  are involved in
  maintaining the
  object group
  abstraction

Server Replica S1
Interoperable Object Group Reference
Profile S1
Profile S3
Profile S2
Server Replica S2
IIOP message
Client
Server Replica S3
18
7. What interfaces are provided to allow
determination of which members are operational,
which are failed, and (where applicable)
which is primary?

• The FaultNotifier interface provides methods that
  are used to disseminate fault information to
  consumers that have subscribed for such
  notifications
• The ObjectGroupManager interface provides the
  get_locations() method, which returns the
  locations of the members
• The primary member of the object group (if
  any) is the first member so designated in
  the list of locations

19
Fault Detection Notification
ReplicationManager
Fault Analyzer
push_structured_event() push_sequence_event()
StructuredPushConsumer
SequencePushConsumer
Fault Notifier
Fault Detector
is_alive()
PullMonitorable
push_structured_fault() push_sequence_fault()
Application Object
20
8. What are the interfaces defined to
control the recovery of entities?

• For the infrastructure-controlled
  ConsistencyStyle, the recovery of objects is
  automatic and uses the Checkpointable and
  Updateable interfaces
• For the application-controlled ConsistencyStyle,
  the recovery of objects is the responsibility of
  the application and no interface is provided

21
Logging Recovery Management
Logging for Active Replication
CORBA ORB
CORBA ORB
CORBA ORB
Logging Mechanism
22
Logging Recovery Management
Logging for Warm Passive Replication
CORBA ORB
CORBA ORB
CORBA ORB
Logging Mechanism
23
Logging Recovery Management
Logging for Cold Passive Replication
24
1. What interfaces are provided to allow the
constitution of active entity groups?
Optional Requirements

• Continuous operation will be achieved using
  object groups having the ACTIVE or
  ACTIVE_WITH_VOTING ReplicationStyles

Active
Active
Client
Request
Active
25
2. What interfaces allow collection of
statistics on faults and provide reports on
these statistics?

• The FaultNotifier interface allows a statistics
  gathering component of the system to register
  as a consumer of fault notifications, upon which
  it can base its statistics
• No interface that provides such statistics is
  defined by the proposal, because such
  statistics are specific to the application

26
3. What framework is provided for classifying
faults, resources, and binding of entities to
resources?

• The proposal defines the ObjectCrashFault, which
  can be extended to include other kinds of faults
  
• The binding between members of an object
  group and resources is provided by the
  definition of locations and of factories
  associated with those locations

27
4. What interfaces are defined for fault
detection and notification that allow group
survivors or a client to perform fault
management?

• The PullMonitorable interface provides for
  detection of faults
• The FaultNotifier interface provides for
  notification of faults to the object group
  survivors or a client

28
5. How can a third party fault manager be
built using the proposed interfaces?

• The specification allows a third party fault
  manager to register as a consumer of fault
  notifications and to manage the membership of an
  object group that has the application-controlled
  MembershipStyle

29
6. What interfaces allow multiple responses
to be suppressed but also allow all
responses to be selectively received so that
they can be voted?

• The suppression of multiple responses is
  automatic
• No interface is provided to allow a client to
  receive all responses, but
• Provision is made for voting on responses
  by the infrastructure in a future extension of
  Fault Tolerant CORBA

30
7. What interfaces allow for the suppression
of redundant requests?

• For IIOP, the REQUEST service context may be
  included in a client's request message to allow
  redundant requests to be recognized and
  suppressed
• If a multicast group communication protocol is
  used, then it is the responsibility of the vendor
  to provide duplicate detection and suppression

31
8. What interfaces are defined to ensure that
a failed redundant identity is effectively
terminated without harmful side-effects?

• The GenericFactory interface provides the
  delete_object() method
• The ObjectGroupManager interface provides the
  remove_member() method