Clustering Implementation in JBoss

1
Clustering Implementation in JBoss

• Jakša Vuckovic
• Università di Bologna

2
Clustering J2EE

• Replicating Resources
• Communication
• Concurrency Control
• Failure Handling
• Controlling Access to External Resources

3
Replicating Resources

• Stateful Session Bean Replication
• Entity Bean Replication
• HTTP Session Replication
• JNDI Tree Replication

4
Sticky Sessions and Homogeneous Deployment

• JBoss staff recommends the use of sticky
  sessions and homogeneous deployment
• Using sticky sessions means that all requests of
  a client arrive to the same node during a session
• Homogeneous deployment means that each
  application component is replicated an all the
  nodes, so requests dont need to span on
  different nodes
• Sticky sessions homogeneous deployment imply
  that each request will be entirely processed on
  one node

5
Stateful Session Beans

• Each instance of a SFSB is associated to a client
• The lifetime of a SFSB is determined by a timeout
  from the last client access
• EJBs cannot generate threads
• All access to a SFSB instance should be
  serialized
• SFSBs are non-reentrant

6
SFSB Replication

• After each invocation of a SFSB, if the bean
  changed, a message is broadcast to the cluster
• The message contains the serialized state of the
  session bean
• Before broadcasting the message a distributed
  lock is acquired
• The broadcast operation is synchronous (blocking)
  with acknowledgements from all the nodes of the
  cluster

7
RMI Clients

• The clients downloads an intelligent stub from
  the server and uses it to access the bean
• The stub encapsulates the load balancing policy
  and the failover mechanism
• The stubs maintains a list of server addresses
  which is updated on each request

8
RMI Clients - Load Balancing

• JBoss provides two basic load balance policies
• RoundRobin
• FirstAvailable (Sticky Sessions)
• These Policies can be applied both to
• Home Interface (Instantiating the bean)
• Remote Interface (Invoking the bean methods)

9
RMI Clients - Failover

• If a call to a node fails the client stub fails
  over to another server or raises an exception to
  the client application
• The stub fails over to another server when
• there is a communication exception
• The server does not responds because it crashed
  or a network partition occurred
• The stub reports an exception when
• There are no more servers to fail over to
• The server reported a GenericClusteringException
  with the flag COMPLETED_YES or COMPLETED_MAYBE

10
RMI Clients - Failover

• If a client makes a request to a node and does
  not receive the response four scenarios are
  possible
• The node crashed before receiving the request
  failover safe
• The node received the request but crashed before
  broadcasting the state failover safe
• The node received and processed the request,
  broadcast the state and then crashed before
  sending the reply the failover node must
  recognize a duplicate request
• The client indicates in the request message if it
  is a failover or the original request

11
Partial Replication Problem

• We have two beans S1 and S2 on two nodes A and B

node B
S1
S2
S2
Client
node A
S1
S2
S2
S1
12
Partial Replication Problem

• The client invokes S1 on node A

node B
S1
S2
S2
Client
node A
S1
S2
S2
S1
13
Partial Replication Problem

• S1 invokes S2

node B
S1
S2
S2
Client
node A
S1
S2
S2
S1
14
Partial Replication Problem

• Invocation of S2 is completed and its state is
  replicated on node B

node B
S1
S2
S2
Client
node A
S1
S2
S2
S1
15
Partial Replication Problem

• S1 tries to make another call to S2 but node A
  crashes

node B
S1
S2
S2
Client
node A
S1
S2
S2
S1
16
Partial Replication Problem

• Now we have an inconsistent state in node B

node B
S1
S2
Client
17
Partial Replication Problem

• There are two possible solutions to this problem
• A replication aware Transaction Manager would
  need to roll back the state of S2 in B also
• The updated state is broadcast to other nodes
  only when committing the whole transaction

18
Network Partitions

• A network failure could divide the cluster in two
  or more partitions that cannot communicate

Node A
Node B
No shared data (Entity Beans, DB) is accessed!!!
Node C
Node D
19
Network Partitions

• The client stub maintains a list of available
  servers
• If a client is not able to contact the server
  instance it is bound to, it tries to contact the
  next from the list until it reaches an instance
  that is its own partition

Node A
Node B
No shared data (Entity Beans, DB) is accessed!!!
Node C
Node D
client
20
Network Partitions

• The client stub maintains a list of available
  servers
• If a client is not able to contact the server
  instance it is bound to, it tries to contact the
  next from the list until it reaches an instance
  that is its own partition

Node A
Node B
No shared data (Entity Beans, DB) is accessed!!!
Node C
Node D
client
21
Network Partitions

• The client stub maintains a list of available
  servers
• If a client is not able to contact the server
  instance it is bound to, it tries to contact the
  next from the list until it reaches an instance
  that is its own partition

Node A
Node B
No shared data (Entity Beans, DB) is accessed!!!
Node C
Node D
client
22
Network Partitions

• Each SFSB is associated to a client, so there
  will be no other clients accessing the instance
  on another partition

A
B
s1
s1
Client 1
s2
s2
s3
s3
No shared data (Entity Beans, DB) is accessed!!!
C
D
s1
s1
Client 2
s2
s2
s3
s3
23
Network Partitions

• Each clients list of available servers will be
  updated to contain only the nodes on its
  partition

A
B
s1
s1
Client 1
s2
s2
s3
s3
No shared data (Entity Beans, DB) is accessed!!!
C
D
s1
s1
Client 2
s2
s2
s3
s3
24
Network Partitions Merging

• When the cluster merges the clients will still
  have the old list of nodes

A
B
s1
s1
Client 1
s2
s2
s3
s3
No shared data (Entity Beans, DB) is accessed!!!
C
D
s1
s1
Client 2
s2
s2
s3
s3
25
Network Partitions Merging

• Only when the client makes the next request, the
  bean gets replicated on the other nodes and the
  clients list is updated

A
B
s1
s1
Client 1
s2
s2
s3
s3
No shared data (Entity Beans, DB) is accessed!!!
C
D
s1
s1
Client 2
s2
s2
s3
s3
26
Network Partitions Merging

• S3 is not updated so the client must maintain a
  separate list of nodes for each SFSB

A
B
s1
s1
Client 1
s2
s2
s3
s3
No shared data (Entity Beans, DB) is accessed!!!
C
D
s1
s1
Client 2
s2
s2
s3
s3
27
Network Partitions Merging Problem

• Suppose node C crashes now

A
B
s1
s1
Client 1
s2
s2
s3
s3
No shared data (Entity Beans, DB) is accessed!!!
D
s1
Client 2
s2
s3
28
Network Partitions Merging Problem

• Client 2 wants to invoke s2 and fails over to
  node A

A
B
s1
s1
Client 1
s2
s2
s3
s3
No shared data (Entity Beans, DB) is accessed!!!
D
s1
Client 2
s2
s3
29
Network Partitions Merging Problem

• If s2 invokes s3 we find an out of date state
  on this node

A
B
s1
s1
Client 1
s2
s2
s3
s3
No shared data (Entity Beans, DB) is accessed!!!
D
s1
Client 2
s2
s3
30
Entity Beans

• Entity Beans are persistent
• Unlike Session Beans, Entity Beans are not
  associated to clients
• Can be accessed by multiple clients, but not
  concurrently
• If an Entity Bean is declared as reentrant it can
  be accessed concurrently but only from the same
  transaction

31
Entity Beans

• JBoss does not replicate Entity Beans,
  replication is left to the database tier
• JBoss controls concurrent access to the database
• All JBoss nodes share the same database
• It is transparent to JBoss if the database is
  clustered or not

32
Entity Beans

• Entity Bean state is loaded from the DB before
  each business method invocation and stored after
  the invocation (Commit option B and C)
• To each J2EE transaction is associated a DB
  transaction
• There are to policies to deal with concurrency
• Optimistic
• Pessimistic

33
Optimistic Policy

• Transactions are executed concurrently
• Each Transaction has its own instance of the
  Entity Bean
• Shared row locking is used in the database
• Before committing the container verifies if the
  Bean changed in the database
• If yes the transaction is rolled back and
  re-executed
• If not the new value is stored in the DB

34
Pessimistic Policy

• Transactions are not executed concurrently
• Exclusive Row locking at the DB is used
• A transaction blocks if it tries to access a
  locked row until the row is released
• A Distributed Deadlock Detection algorithm is
  used to detect Deadlocks

35
Entity Bean Replication Failures

• If the node owner of a transaction crashes the DB
  rolls back the transaction
• Network partitions must be handled by the DB
  clustering mechanism
• With a non replicated database, the approach of a
  primary partition is used where the primary
  partition is defined to be the one containing the
  database

36
Entity Bean Replication

• Relies intensively on DB
• Maximizes DB I/O
• If we dont want a single point of failure, we
  must use a replicated database
• Entity Beans are not used as cache
• A distributed cache with distributed locks is
  under development

37
HTTP Session Replication

• JBoss is able to work with two Serlvet Containers
• Jetty
• Tomcat
• With both containers we have three replication
  schemes
• Instant Snapshotting HTTP Sessions are
  replicated after each HTTP request
• Interval Snapshotting HTTP Sessions are
  replicated on a time based interval
• Economic Snapshotting (JBoss 3.2) HTTP Sessions
  are replicated only if setAttribute() has been
  called

38
HTTP Session Replication

• An HTTP Session is implemented as a serializable
  object
• The HTTP Session state is NOT broadcast to the
  other nodes but replication is implemented
  through an Entity Bean
• The HTTP Session is a cmp-field of that bean
• The CMP engine serializes the HTTP Session object
  along with its contents
• The HTTP Session contains also references (remote
  objects) of SFSBs
• SFSBs do not get serialized with the HTTP Session

39
HTTP Session Replication

• Failures are handled by the EJB Replication
  mechanism
• Client fail over behavior depends on the HTTP
  dispatcher
• A JBoss HTTP Dispatcher is under development

40
JNDI Tree Replication

• Each node has a local JNDI Tree
• All nodes share a replicated HA-JNDI Tree
• When a client looks up an object on node N
• The replicated HA-JNDI service is invoked first
• If it cannot find it there it looks in the local
  JNDI Tree
• If its not there it asks all other nodes if they
  have it in their local JNDI Tree
• If it hasnt been found a NameNotFoundException
  is thrown

41
JNDI Tree Replication

• Unlike clients, EJBs access only the local JNDI
  Tree
• This solution was chosen because of
• Compatibility with existing applications
• Distinction between local and replicated objects
• Low network traffic in homogeneous clusters

42
Distributed Cache

• Deadline for June (JavaONE)
• Based on JavaGroups
• Will be a Jboss Service
• Highly Configurable
• Will be applied to Entity Bean Clustering and
  Session Bean Clustering
• Will enable replication of the Database

43
Distributed Cache Semantics

• Three semantics
• Asynchronous
• broadcast update and return immediately
• Synchronous
• broadcast update and wait for acknowledgements
• Serialized Synchronous
• acquires locks before updating a bean
• ensures ordering but not consistency
• consistency is obtained at another level

44
Cache Contents

• The cache will implement the XAResource interface
• This will enable the use of the cache in a Two
  Phase Commit protocol
• The mapping of the XAResource semantics to the
  cluster is not a trivial problem
• This could enable to do the DB update, cluster
  replication and sending the response to the
  client an atomic operation

45
Conclusions

• JBoss clustering algorithm does not handle all
  failure patterns correctly
• Simply changing the GC semantics will not solve
  the problems
• To make the existing mechanism work correctly
• A replication aware Transaction Manager needs to
  be implemented
• During state merging the out-of-date beans must
  be marked
• An alternative would be to re-implement the
  clustering from scratch
• The development of the Distributed Cache by JBoss
  authors is a good opportunity to redefine the
  clustering algorithm